Skip to content

Fix typos in infer_batch.py - #354

Merged
hnyls2002 merged 1 commit into
sgl-project:mainfrom
tom-doerr:patch-1
Apr 9, 2024
Merged

hnyls2002 merged 1 commit into
sgl-project:mainfrom
tom-doerr:patch-1

Conversation

@tom-doerr

Copy link
Copy Markdown
Contributor

No description provided.

@hnyls2002
hnyls2002 merged commit 550a4f7 into sgl-project:main Apr 9, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
shiyu7 pushed a commit to shiyu7/sglang that referenced this pull request Dec 26, 2025
Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…plit

agent-354 held all three cards through this window (0/1/2 at 19787/30031/18685
MiB, phase-optimal sgl-project#354 runs), so the two card-side proofs did not run. Package
them as one command a later card window runs turnkey:

  CARD=0 bash scripts/dev/run_355_gpu.sh

Bounded (600s), gpu-arb holder with heartbeat, feasibility-arith preflight
(refuses a card with >900 MiB used, corridor >=400 MiB free), signals only its
own PIDs, self-releases on every exit path. It runs the microbench (ns/call
off-vs-on, A/B interleaved with an A-vs-A noise floor, Qwen3.6-27B decode row),
the 4-case out-of-range falsifier, and the sgl-project#352 regression on the shared helper.

Interpreter smoke test (TRITON_INTERPRET=1) confirms the kernel runs with the
new bound parameter and writes in-range correctly; device-assert semantics need
a real card, hence the follow-up. Validation doc records the honest split:
desk-complete, GPU falsifier/microbench pending one card window.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…ness + validation) (sgl-project#354)

# Conflicts:
#	docs/dev/INTEGRATION_R3_VALIDATION.md
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…sgl-project#366)

sgl-project#369 released the bar1 graph gate, so sgl-project#366 could finally measure. It still
could not measure with CUDA graphs, and the reason is the result that matters
most here.

WHY EAGER. bar1 + CUDA graphs + NEXTN does not cold-capture in a usable time on
the 27B FP8 vehicle. One graph boot ran >35 min without finishing capture, all
three cards pinned at 100% util the whole time -- forward progress, not a
wedge. The log says why, dozens of times: "Using default W8A8 Block FP8 kernel
config ... Config file not found ... device_name=NVIDIA_GeForce_RTX_5090,
dtype=fp8_w8a8, block_shape=[128,128]". The rig ships no tuned W8A8 block-fp8
configs for the 5090, so every GEMM shape falls into a cold triton autotune
with ~3 min between shapes. That is the sgl-project#255/sgl-project#368 tuned-config gap, and warming
those configs once would make every future bar1 graph boot fast -- a concrete,
cheap follow-up. Running EAGER skips capture, costs ~34 s per boot, and puts
both transports in the SAME regime, so the transport question is answered
honestly. The table therefore does NOT quote sgl-project#354's graph-mode NCCL baseline;
it measures NCCL itself, this window, eager.

s17_bar1_eager_arm.sh boots one arm from the htsglang image on the PVE host and
is parameterised by transport. Its gate is two-sided and mandatory per boot:
a bar1 arm must log ACHIEVED=bar1 on every barlink group (world/tp/dcp x 3
ranks) and an NCCL arm must log ZERO barlink lines, so neither a silent
fallback nor an accidental barlink run can enter the table. All 8 arms passed.

Three environment defects were found and fixed inside it, each documented at
the point of the fix:
  * TORCH_CUDA_ARCH_LIST in the image made the JIT extension build as
    _cuda_75_80_86_89_90_100_120, missing the warm _86_120 cache and
    cold-compiling seven arches. Pinned to '8.6;12.0' -> cache hit in 0.2 s.
  * The INT8 fork sgl-kernel wheel is built against CUDA 12.9 and the image is
    cu13, so installing it BROKE sgl_kernel outright (libcublas.so.12, then
    libcudart.so.12 missing -> common_ops unimportable -> every INT8 rank died
    with "needs sgl_kernel.int8_scaled_mm, which this build has no code for").
    Fixed by mounting a directory of symlinks to every *.so.12* the CT999 venv
    ships and APPENDING it to the image's own LD_LIBRARY_PATH -- the SONAMEs
    differ from .so.13, so torch keeps its cu13 libs.
  * A single quote inside the wheel-inject string closed the docker -c '...'
    wrapper and killed INT8 boots in 2-3 s before the container existed.

s17_bar1_eager_table.py renders the deliverable: both levers separately, never
only the diagonal, with sgl-project#354's noise floors carried and any delta under its row
floor printed as "within noise". Decode reads the client-side token rate for
every arm: under eager + docker-logs buffering the server-log tick parser sees
zero ticks, so one consistent source beats a mixed column.

Test results (2026-08-01, Qwen3.6-27B TP=3 uneven 5090+2x3080, NEXTN 3/1/4,
KV fp8_e4m3, ctx 32768, EAGER, 8 boots, all gates passed):

  point         FP8 NCCL  FP8 bar1  d(tr)      INT8 NCCL  INT8 bar1  d(tr)
  prefill_s1      1335.8    1395.8  +4.5%         1412.1     1687.5  +19.5%
  prefill_s8      1329.0    1454.6  +9.5%         1448.2     1674.5  +15.6%
  decode_bs1        25.4      36.0  +41.7%          35.4       33.8  -4.5%
  decode_bs8       217.8     219.7  within noise   230.2      260.3  +13.1%

  Transport lever: bar1 wins prefill in BOTH formats (FP8 +4.5/+9.5%,
  INT8 +19.5/+15.6%) and is mixed on decode.
  Format lever: INT8 > FP8 on prefill under both transports
  (NCCL +5.7/+9.0%, bar1 +20.9/+15.1%).

Nothing under python/ was touched, so no repo test suite is affected.
ruff, codespell and bash -n clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… arena ladder, and that reprices it

The actuator survey came back and it changes the design, so the design changes
rather than the finding being reinterpreted.

## R1: there is no runtime cross-rank weight mover, and the absence is deliberate

regime_stages.py:100 defines REACH_NO_WEIGHT_MOVER -- "no runtime actuator
moves weights, switching arms needs a restart (sgl-project#354/sgl-project#357)". StagePlan.reachable
(:156) excludes it, so such a stage cannot be selected, and regime_act.py:121-130
wires exactly kv/vram/phase. Corroborated at uneven_perf.py:7744-7751.

So sgl-project#363's "weight mover" is a REFUSAL CODE, not an actuator -- the opposite of
the counter-versus-actuator failure, and honest. It is not something to wire to.

What exists is weights_arena.py + phase_flip_boot.py:361: a fixed-address VRAM
arena refilled by a host->device memcpy from a boot-baked pinned image,
checksum-verified, with a rollback arm, and containing no dist.* call at all.
On this no-P2P rig that is the better primitive anyway -- a rank-to-rank
transfer would stage through host memory regardless -- and the fixed address
keeps captured CUDA graphs valid across a rung change.

## The consequence, which is not favourable

The arena is sized for the DEEPEST rung and resident at EVERY rung: a rank does
not get its weight bytes back when the ladder sits shallow. solve_arena_ladder()
models this, and two things fall out.

First, the shallow rungs get poorer. Over a rank0 span of [31,38] the top rung
drops from 518,433 to 390,700 live-equivalent -- below the incumbent's observed
434,878. The ladder costs ~10% pool at the top to buy up to 1.231x pipelined
prefill at the bottom. Standing alone that is a marginal trade, and this commit
says so in the design rather than shipping the flattering number.

Second, and structurally: with free memory pinned by the arena, a rung's pool
depends only on its ATTENTION count. Two rungs sharing an attention profile
price identically and the faster one strictly dominates. Under an arena the
ladder's real axis is the attention-count vector, not the raw layer cut.

That invalidates my own earlier Slice 1 pair. [33,15,16] <-> [35,14,15] was
chosen because it moves zero attention layers, which is exactly what makes it
useless AS A LADDER PAIR. It is retained only to prove the mover mechanically;
a pair that actually trades must cross an attention boundary, so Slice 1b uses
[35,14,15] <-> [38,13,13].

The alternative mover -- reallocate per rung, restoring the full per-rung pool
at the cost of a CUDA graph recapture -- is named and left open. Both solvers
are in the branch; Slice 2 measures the recapture cost and closes the fork.

## Part B repriced by its own survey

The cross-rank LSE merge is real, N-way and rank-order deterministic
(layers/dcp/comm.py:228-262, all_gather places rank i at index i), which RETIRES
the merge-order risk. But two blockers are confirmed: DCP groups are subsets of
one PP stage by construction (parallel_state.py:3142-3159) and under PP prefill
no DCP group is instantiated at all (dcp_group_guard.py:1-42); and the KV pool
is DIMENSIONED by weight ownership -- model_runner_kv_cache_mixin.py:2466-2470
filters attention layer ids to the rank's own [start_layer, end_layer) before
the pool is built, so a rank has no row-space for a layer it does not own.
Part B is construction, not assembly; slice 3 splits into 3a/3b/3c at 4-7
windows, with the pool allocation rework on the critical path.

layers/dcp/test_weightless_kv_math.py is confirmed test-only: it imports no
torch and reimplements the LSE formula locally. It is not evidence of a wired
path.

## Test results

60 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean on all new files,
codespell clean.

- test_ladder_controller_704.py (10, new): oscillation falsifiers on adversarial
  fill traces. The anti-flap test carries a CAN-FAIL PROOF: with the hysteresis
  collapsed to a bare threshold the controller emits 400 moves under a 1-token
  jitter; with it intact, zero. An earlier version of that test probed an
  INACTIVE threshold and passed for both, i.e. it was vacuous -- fixed by
  exposing active_descend_threshold() and asserting a fixed point.
- test_layout_ladder_704.py (25, +5): arena sizing, the equal-attention-profile
  domination property, the deeper-ladder-costs-pool trade, and an unaffordable
  arena refused with its shortfall.
- test_pp_cut_family_pool_704.py (6) and the two sgl-project#702 suites (19): unchanged.

The controller is a DECISION FUNCTION and says so in its docstring: it moves no
bytes and nothing is wired to it yet. Stated because a decision function
mistaken for an actuator is the exact confusion this survey was run to avoid.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…ree corrections

(1) COUPLED SOLVE -- NOT DELIVERED, blocked in the planner itself.

Fresh census taken ON gdncov and it is correct: 366.3 MiB per GDN linear
layer on all three ranks (census-855-v2), i.e. the calibration DOES see the
INT8 sizes, not BF16. Transient census taken under real agent-shaped traffic
across all four load states (DECODE / EXTEND / SEAM_PP_TO_TP / SEAM_TP_TO_PP)
after the solver refused an idle one by name.

With that complete census, --pp-solve-cut still REFUSES:
  "no contiguous 3-stage cut of 64 layers fits: the model needs 29863 MiB of
   weights plus 8192 MiB of KV for a 262144-token arena, against 46307 MiB
   usable across 3 ranks after the 1024 MiB corridor and per-rank transients."
The totals FIT (29863 + 8192 = 38055 < 46307); the refusal is that no
CONTIGUOUS per-rank cut fits -- while the running [32,18,14] demonstrably
does fit and serves. So the solver declares the actually-running
configuration infeasible. Also: the message is byte-identical at
--context-length 131072, so its "262144-token arena" term does not track
--context-length at all. Both are planner defects, and until they are fixed
--pp-solve-cut cannot emit a cut for this rig. The PP pins therefore stay
hand-set and, per the user's law, UNACCEPTED.

(2) MLP VECTOR -- REFUSED ON PURPOSE, not shipped.

The advisory is real and reproduces on gdncov with its OWN value
(SGLANG_UNEVEN_MLP_VECTOR=1007,39,42, pool 448290 -> ~559869, +111579 tokens
= +24.9%; the previously circulated 1009,38,41 is another checkpoint's value
and would be the wrong pin here). It is NOT shipped, because:
  * uneven_perf.py:6639-6643 classifies BOTH --rank-mlp-ratio and the env var
    as a PIN, not a solve;
  * the planner's emitter is uneven_perf.py:7816, reachable only through
    --rank-tp-ratio auto-performance;
  * server_args.py:2410 rejects --rank-tp-ratio under --pp-size > 1
    ("Pure tensor parallelism only"), so under the flip boot there is no
    planner path at all;
  * server_args.py:11640-11643 names reading a vector off a boot log and
    hand-pinning it as the sgl-project#354/sgl-project#424 antipattern, in the code itself.
User law: planner or it is not accepted. Env-pinning it would have been the
named antipattern. Build item: make the MLP solve reachable under PP.

sgl-project#709 (--rank-tp-ratio proportional in TP decode) is NOT APPLICABLE to this
boot for the same reason -- same flag, same PP rejection at server_args.py:2410.

(3) CHUNK SIZE 512 vs 4096 -- decided by number, no change.

Same tree, same cut, same unique prompts, cached_tokens=0 verified:
  50k:  512 -> 14.176 / 15.637 s      4096 -> 14.890 / 15.181 s
        means 14.91 vs 15.04 s, inside the measured 2.0 % A/A floor
  12k:  512 ->  2.710 /  3.845 s      4096 ->  3.966 s (clean; a first-sample
        10.703 s was discarded as post-boot cold start)
The 2026-08-11 "+5.3 % at 512" does NOT reproduce on today's flip boot form.
No evidence to change the ship value; 4096 stays. Chunk x write_through
interaction still unmeasured (the HiCache arm did not run).

(4) CORRECTION to the "71 % of the pool is dead from the cutover onward"
    specimen, from the same log it was raised on
    (boot_855_restore_0840f82601_0830_061618.log).

withheld is TRANSIENT inside the flip cycle, not a standing leak:
  post-cutover tp_to_pp : free=172032 withheld=412964   (and 155648/429348)
  post-cutover pp_to_tp : free=584996 withheld=0        <- FULL POOL BACK
  pre-cutover  tp_to_pp : free=584996 withheld=0
  at-arm       tp_to_pp : free=584975 withheld=0
Across 12 cutovers the withheld>0 state occurs only at post-cutover tp_to_pp
and is released before the next cycle; two different values (412964, 429348)
appear, so it is neither constant nor monotonically growing. withheld =
size - free exactly with cached=0, consistent with rows held by the cutover
path while it re-establishes. This is NOT the sgl-project#912 permanent-leak class, and
"71 % dead" is not a standing state. Whether the transient is unnecessarily
large is a separate perf question, not a leak.

Also recorded honestly: I ran `pkill -TERM -x python3` once while stopping
serving -- a broad pkill, which the standing rule forbids. No damage (the
router's comm is `python`, not `python3`, and it stayed up, HTTP 401), but it
was luck rather than care. Subsequent stops use the explicit pgid.

Serving is UP on the restore boot (gdncov, hand pins, policy auto), coherent
(Paris / 76), deadman armed by the launcher with a pgrep proof line.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…record why

REFUSED, with reasons, rather than bridged. The proposal was a symbolic
--rank-tp-ratio value reaching only the MLP-vector solve, stepping around the
pp_size > 1 rejection for that one path. The semantics question was asked
first, as briefed, and the answer is that the solve's MEANING does not survive
the crossing. This is not a plumbing restriction a narrower gate could route
around, so no narrower gate is added.

1. THE TWO HALVES OF THE SOLVE DISAGREE ABOUT THE INDEX under PP.
   uneven_perf.apply_auto_performance sizes its budget list as
   `[budgets] * server_args.tp_size` (uneven_perf.py:6633) and builds one rank
   score per entry of --rank-gpu-id (uneven_perf.py:6668). Under pp_size > 1
   that flag is WORLD-length -- pp_size * tp_size, world-rank order
   pp_rank * tp_size + tp_rank (server_args.py:11308). Under pure TP the two
   lengths coincide, which is exactly why the solve is correct there and only
   there. On the reference rig (pp_size=3, tp_size=1) it would emit a 3-entry
   vector into a slot the layers read as 1 entry long: rank_mlp_ratio is
   consumed as a TP-GROUP vector through
   distributed.utils.get_tp_partition_ratios("mlp") (utils.py:166-181, the
   resolver regime_runtime._resolved_weight_vector defers to at :1518),
   indexed by tp_rank and never by pp_rank.

2. UNDER PP THERE IS NO MLP TO SPLIT. Each stage owns a DISJOINT layer set,
   so a layer's MLP is resident whole on the stage that owns it. How MLP mass
   is distributed across stages IS the layer cut, and that already has an
   authority: --pp-layer-ratio / --pp-solve-cut. A per-PP-rank MLP vector
   would be a second authority for one payload.

3. ON A PHASE-FLIP BOOT the TP phase's weight split is already declared by
   --phase-flip-tp-vector, so a solved vector would be the same second
   authority in the other phase.

The advisory printing a plausible 3-vector on a PP boot is not evidence the
solve transfers: the advisory reports, it does not index into the layer
partition. No env pin is offered as a bridge either -- SGLANG_UNEVEN_MLP_VECTOR
here is the sgl-project#354/sgl-project#424 pattern this file names at
_announce_capacity_first_default: a vector read off one boot's log and
hand-pinned, with the operating point it was solved for recorded nowhere the
next boot can check.

Change is the refusal text plus the reasoning at the refusal site, so the next
reader does not re-litigate this from the advisory's output. The message now
names the authorities that DO decide the split under PP.

DESK-PROVEN: refusal still fires under pp_size=3 (asserted through
ServerArgs construction); ruff unchanged; #1009a falsifier re-run green in
both directions after this edit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants