Skip to content

Fix flashinfer - #700

Merged
Ying1123 merged 2 commits into
mainfrom
ying-fix
Jul 23, 2024
Merged

Ying1123 merged 2 commits into
mainfrom
ying-fix

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

@Ying1123
Ying1123 merged commit cf99eab into main Jul 23, 2024
@Ying1123
Ying1123 deleted the ying-fix branch July 23, 2024 08:27
Ying1123 added a commit that referenced this pull request Jul 23, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
* Update test_utils.py

* Update test_openai_server_ignore_eos.py

* Update run_suite.py

* Update run_suite.py
hhhh1252023 pushed a commit to hhhh1252023/sglang_public that referenced this pull request Jun 6, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
… percent at L=16

Desk halves of sgl-project#700. Paused mid-ticket for the sgl-project#702 priority insert; part 3
(the identity arm) is specified but no arm has been run.

(1) Wiring contradiction resolved: the KERNEL HEADER is stale, not the wiring.
fused_recurrent_linear_replayssm.py:10-12 claims the feature is "NOT yet wired
into the memory pool / radix cache / scheduler / backend dispatch". It is wired
in all four: ring allocation and documented layout (memory_pool.py:469-477,
611-633), cursor reset on slot alloc (:1530), copy_from with a stated invariant
and dst reset (:941-968), flush/clear (:908), write_pos carried in the
offload/restore blob (:1017, :1049, :1075), the radix donate capped to the last
flush boundary (mamba_radix_cache.py:599-608), its unified-cache mirror
(mamba_component.py:409-417), backend plus CUDA-graph static buffers
(hybrid_linear_attn_backend.py, 54 refs), and dispatch (gdn_backend.py:497-537).

So sgl-project#700 does NOT close as refused-until-fixed: the seam hazard the ticket
anticipated -- a donate or copy mid-window against an unflushed ring -- is
addressed head-on and documented at both sites. No correctness hole found.

Two findings remain. (a) The copy_from invariant is enforced only under
debug_memory_pool, off by default, so in production it rests on caller
discipline in a docstring; a fourth caller would violate it silently. That is
the sgl-project#624 drift class -- hardening candidate, not a bug today. (b) _build_tree's
docstring says the pools are built "on CPU", but construction reaches
utils/common.py:1322 and raises "No accelerator ... is available" under
CUDA_VISIBLE_DEVICES="". A second comment that does not match its code.

Execution status, stated plainly: the ticket asked to settle this BY EXECUTION
and that is blocked hermetically by (b) -- the pre-existing
TestFlushResetsMambaPool fails identically, so this is not introduced here. The
determination above is therefore made by reading. The executable proof is
committed and waiting for a GPU window: TestReplaySsmRadixSeam, four tests
covering ring allocation, dst-cursor reset, the can-fail proof that the
documented invariant trips under the debug guard, and one that pins (a) as a
fact by showing an unflushed source is copied silently with the guard off.
THESE TESTS HAVE NEVER RUN AND ARE NOT REPORTED GREEN.

(2) The net count left open in sgl-project#325: total(L) = S + R + ((L-1)/2)R + S/L with
S = 3,145,728 B and ring append R = 32,960 B/step (d 24,576 + k 8,192 + g 192;
note replayssm_k is per K-head, 16, not per V-head). At the default L=16 the
saving is 40.4 percent of chain bytes, a 1.68x reduction -- below the header's
"roughly halved" because the reconstruction re-read is priced at zero there.
The optimum is L* = sqrt(2S/R) = 13.8, so the shipped default sits within 0.1
percentage points of optimal and the curve is flat from L=8 to L=24. L=1 comes
out worse than baseline at -0.5 percent, the sanity result a correct count must
reproduce. Against sgl-project#325's >10 percent gate this clears by 4x versus 0.99
percent for the best fusion.

(3) Identity arm specified, not run: CPU-sampled inputs, A-vs-A floor first,
probe kept under the ~109-token GDN prefill non-determinism ceiling, GDN
scalar-gate only, and the identity result gates whether the perf arm is worth
running at all.

Docs plus one window-gated test class. No enable-flip, no deploy.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…s, not numbers or env flags

Binding directive (PLAN_PERF_PIPELINE_2026-08-16, "PLANNER-SOLVED, UNIVERSAL"):
the family-split verdict must be an objective/constraint set so solve() picks
family ratios on any hardware, and the ReplaySSM enable must be a
planner-visible gated capability rather than a flag someone remembers to set.
The pp_cut phase-rule work is the pattern: rules, not numbers.

planner/family_split.py -- nothing in it names a GPU or a model. Collective
costs arrive from the census instruments, bandwidths from the pair matrix,
capacity from the ledger. The rules:
  * a family's BLOCKING collectives vanish iff its placement is concentrated on
    exactly one rank; partial concentration buys nothing, and DEFERRED
    collectives never vanish, so a family whose collectives are all deferred can
    never repay concentration;
  * sharded time depends on the shard POLICY -- equal width shard lets the
    slowest rank bind, bandwidth-proportional (uneven TP) has every rank finish
    together at bytes / sum(bandwidth);
  * concentration therefore pays only when it escapes a slow BINDING rank;
  * capacity is a hard per-rank constraint, and under the TP sum rule
    concentration redistributes rather than consumes;
  * an unmeasured collective cost is REFUSED, not guessed.

Writing it as rules immediately falsified my own NOTE_705 headline. That note
priced concentration against an equal 1/3 shard and derived a 14.3 us gate, but
this fork ships --rank-tp-ratio, so the honest baseline is the proportional
shard: 1.73 ms instead of 2.51 ms, and the gate roughly DOUBLES to 30.5 us. The
solver derives both and a test pins the pair; the note carries a revision-2
correction block. The measured-cost fixture of 40 us still clears the corrected
bar, so the recommendation is unchanged -- but it was closer than the note
claimed.

The generality test is the point of the exercise: on a uniform-bandwidth rig
the same rules REJECT the placement they recommend here, because there is no
slow rank to escape. A solver that still said "solo" would be fitting this rig.
Capacity conservation, a slower host never being chosen, and the
bandwidth-proportional fallback ratios are pinned alongside.

planner/gated_capability.py -- ReplaySSM's enable becomes a decision with a
printable reason. An UNMEASURED quality gate is a refusal, distinct from a
measured failure ("we did not look" and "we looked and it diverges" support
different decisions). An unmeasured net is a refusal. Byte-identical plus net
above the floor enables; anything not byte-identical is lossy and refused
unless lossy is explicitly permitted, and a large net does not override that
ordering. The live ReplaySSM state -- net counted at 40.4 percent, identity gate
never run -- is pinned as REFUSED, which is exactly where sgl-project#700 left it.

Tests: 18, red first (11 + 7). Planner suite 2611 passed, 2 failed, both the
pre-existing test_rejected_evidence_pins failures verified earlier against clean
integration/r2. ruff clean. No arm, no deploy.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…laySSM gate built

sgl-project#707 CLOSED. The holdback is not a reserve in the intuitive sense and never
was, which is why four attempts to model it as one failed. It is a CAP:

    allowed_tokens = id_space + (free_at_measure - arming_floor - margin)/cell
    adjusted_bytes = allowed_tokens * cell
    holdback_frac  = 1 - allowed_tokens / (profiled_bytes / cell)

from phase_flip_seam_reserve.floor_allowed_tokens -- "largest id space whose
RESTING FREE COLUMN still holds the arming floor". An equality, solved in closed
form, reached through the one funnel every sizing path crosses.

Verified on all three ranks of the instrumented boot: model 45.141 / 44.072 /
60.256 percent against reported 45.143 / 44.074 / 60.258. Agreement 0.002 pp,
the residual being MiB rounding in the logged budget. id_space 435,334 on all
three.

The three questions answered. (1) The percentage is a CONSEQUENCE of capping the
pool so the resting free column still holds the floor, not a parameter anything
computes. (2) The PP pass pays because it is the layout a flip departs from and
must rest above the floor; the TP-stack pass has no flip to arm from and spends
its whole budget -- holdback 0.000 percent on all three ranks confirms. (3) The
BINDER holds back most because both extremes meet on PP2: smallest cell (4 attn
-> 8192 B) gives it the LARGEST raw token capacity of the three (1,097,728),
while its resting free column sits 7.4 MiB above its arming floor -- essentially
on it -- pinning allowed at barely more than id_space. It binds for the same
reason it holds back most, which is why the two always co-occurred and looked
like coincidence.

Retired: all fitted candidates, and the [33,15,16] discriminator boot, which now
has no calibration value. The rank2 blind spot is preserved explicitly because
it still governs TIMING calibration, which this result does not touch.

Handed to Slot-3: rung pools can go from extrapolated to EXACT without booting
each rung. cell is config-derived, id_space/free_at_measure/margin come from the
current seam record, arming_floor from sgl-project#676, and the layout shift is arithmetic
over two established constants -- free_at_measure(cut) = free_at_measure(booted)
- delta_weights (374.2 full-attention / 476.2 linear per layer) - delta_mamba
(51.20 MiB/GDN-layer). What stays extrapolated is only that shift, which is a
far smaller claim than carrying another layout's capture behaviour and should be
labelled as such rather than as measured.

sgl-project#700, both halves built. The wiring contradiction is RESOLVED in-code: the
kernel header's "NOT yet wired into the memory pool / radix cache / scheduler /
backend dispatch" was stale and false in all four places it named, and the
header now carries the file:line evidence for each plus the note that reading it
as "inert" cost sgl-project#325 revision 1 a wrong conclusion. The byte-identity half is
planner/replayssm_identity.py plus scripts/replayssm/identity_probe.py: A-vs-A
floor before A-vs-B, probe capped under the ~109-token GDN determinism ceiling,
CPU-sampled inputs only, GDN scalar gate only, and same-tokens-with-nonzero-delta
classified as NOT byte-identical rather than rounded up. An unrun measurement is
a refusal, never a default-on. The probe refuses before touching a GPU --
verified by running it with 256 tokens.

--enable-linear-replayssm stays OFF: wiring resolved, fidelity unmeasured.

Tests: 11 for the identity gate, red first. Planner suite 2665 passed / 2 failed
-- the pre-existing test_rejected_evidence_pins pair. ruff clean.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…, verdict recorded

Decided without a boot, from the kernel's OWN registered test
(test_linear_replayssm_decode.py), run on GPU2 alongside live serving: 6 passed
/ 36 subtests.

The test settles it by construction. Its docstring states the kernel is
"algebraically equivalent but floating-point REORDERED", and the suite contains
ZERO exact-equality assertions -- every check is tolerance-based, atol 2e-6 at
L=1 and 1e-4/1e-3 at L>=4. ReplaySSM produces a nonzero delta BY DESIGN: the
reordering IS the optimization. So it is not byte-identical and CANNOT be, at
any L, on any rig.

--enable-linear-replayssm stays OFF. The enable path is Quality-Last per
standing doctrine: lossy features go to the back of the queue and require
quality-suite evidence. The byte gate is retired as the criterion, because a
criterion nothing can pass is not a gate -- it is a permanent refusal wearing a
gate's clothes, and leaving it in place would have looked like an open question
forever.

My harness is demoted accordingly, in its own docstring so it cannot be
miscited: planner/replayssm_identity.py and scripts/replayssm/identity_probe.py
are NOT the enable gate. Their remaining value is regression-pinning the
tolerance band -- if a future kernel change widens divergence beyond the
registered envelope that is a real regression, and the classifier's distinction
between "same tokens, nonzero delta" and "tokens changed" is the right shape to
catch it.

The miss, recorded in the note: I built a measurement harness without first
checking whether a registered test already covered the question. It did,
definitively, and more cheaply and sharply than an end-to-end probe could. Same
lesson as the mamba allocation instrument, which also already existed. Twice now
the answer was already in the tree -- look for the existing instrument before
building one.

Docs plus one docstring. Tests unchanged, 11 passed. ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant