Skip to content

Decode Incrementally - #517

Merged
merrymercy merged 8 commits into
mainfrom
fix-decode
Jun 12, 2024
Merged

merrymercy merged 8 commits into
mainfrom
fix-decode

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jun 8, 2024 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

  • Use decode incrementally like vLLM and TGI when there is a jump-forward.
  • Also removed the special judge for whether there is a leading space.
  • Align with outlines, using byte-level FSM, supporting UTF-8 characters such as Chinese when using regex and jump-forward.

@merrymercy merrymercy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

  1. Does it affect the throughput in non-streaming mode?
  2. Can it improve the throughput for streaming mode?

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

LGTM.

  1. Does it affect the throughput in non-streaming mode?
  2. Can it improve the throughput for streaming mode?
  1. It only brings a minor overhead of decoding a few tokens (normally 5) more for each request when it is non-streaming.
  2. Sure.

@hnyls2002
hnyls2002 requested a review from merrymercy June 10, 2024 18:06
@merrymercy
merrymercy merged commit 9c902b1 into main Jun 12, 2024
@merrymercy
merrymercy deleted the fix-decode branch June 12, 2024 06:39
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026
* improving code readability for http server
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… correct the §1.3 model, and make the sgl-project#431 guard non-blocking

Two commits:

test(sgl-project#483,sgl-project#459): sgl-project#483 ("check_regressions prices prefill on stale rates")
is REFUTED -- check_regressions already prices on the RESOLVED rates
(landed with sgl-project#475). What was actually missing was the PIN: the only test
calling the function is checkpoint-gated and skips in every hermetic run,
so a revert would be invisible. test_check_regressions_pricing_483.py is
the hermetic falsifier. Also makes the s12 harness honest: an absent
measurement is None, never 0.0 (a spec-off tick has no accept length, and
0.0 silently survives every arithmetic it enters); and any procedural
property a verdict rests on (warm-up discarded, draws back-to-back, gap
between draws) is recorded as a measured number in the artifact, not taken
on the harness's word -- the s12 warm-up draw ran for months and was
discarded silently, so a warmed-up point and a cold one wrote the same
file.

perf(sgl-project#517): names the actual decode-loop BAR1 collectives and corrects the
catalog's round model -- a decode round is NOT free of host-path
collectives (the belief the old model rested on). Per NEXTN round there
are 3 replay boundaries (the draft chain is one captured graph, not one
per step) and 5 host-path BAR1 broadcasts, all sgl-project#50 speculative
rank-agreement syncs, reaching barlink because a barlink boot never
constructs pynccl. Measured cost: the sgl-project#431 blocking status-word read costs
-9.22% decode TPS against the same-tree NCCL baseline; removing it
recovers +2.68%. Fix: the abort-guard status word is now read
asynchronously (non-blocking D2H + cudaEventQuery, returning what an
earlier check staged), trading reporting LATENCY for zero stream
synchronization per check -- detection is unaffected because ctlStatus is
sticky. SGLANG_BARLINK_BAR1_ABORT_MAX_LAG (default 4) bounds the
deferral with a can-fail proof; ..._ABORT_DEFER=0 restores the exact
pre-sgl-project#517 blocking behavior. NOTE_517_bar1_guard_desk.md has the full
model, desk-only (no GPU arm run yet -- ticketed).

Two real conflicts, both resolved by combining rather than choosing a
side:
- FEATURE_CATALOG.md §12 (same insertion-point pattern as every merge
  today): HEAD's "Verified carve-outs (sgl-project#514)" paragraph kept, this
  branch's two sgl-project#459 paragraphs ("absent measurement is None" /
  "procedural property must be in the artifact") appended after. Two
  OTHER hunks (the sgl-project#483-REFUTED note near §2, the "Guard cost" note near
  §7) landed with no conflict at all -- neither region had been touched
  since this branch's base.
- test/registered/unit/distributed/_bar1_marker_source.py: both HEAD (via
  sgl-project#514's earlier edits) and this branch carry DIFFERENT LINE_* pins into
  barlink.py/barlink_bar1.py, because each was written against its own
  version of those files. Neither side's numbers are correct once both
  sets of source edits are combined. Resolved by reading the ACTUAL
  post-merge source: grepped the real logger.info/logger.warning/raise
  RuntimeError call-start lines in the merged barlink.py and
  barlink_bar1.py (2236/2250/3207/810) and verified all four renderers
  execute correctly against them before committing -- not a textual
  merge of the two guesses.

All 17 catalog sections and every other additive paragraph verified
present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The five directly touched/new test files (_bar1_marker_source.py's
  renderers verified live against the real merged source, plus
  test_barlink_bar1_abort_431.py, test_barlink_bar1_abort_deferred_517.py,
  test_s12_harness_459.py, test_check_regressions_pricing_483.py): 67 + 7
  passed, 0 failed.
- test_s12_log_analyse.py specifically (s12_log_analyse.py is touched):
  same 4 pre-existing failures, unchanged (fixture-data-dependent, not
  source-code-dependent).
- test/registered/unit/distributed/ (full): 24 failed, 2206 passed,
  8 skipped, 723 subtests -- same 24 failing IDs as the established
  baseline, byte-identical.
- ruff check --select=F401,F821,UP037: clean.
- codespell: 7 hits, all confirmed pre-existing (a German prose sample
  string already in scripts/gpu_battery/s12_prefill_kurve.py and
  s14_decode_punkt.py at this branch's base, untouched by its diff).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… medians, honest server state

Three new server-side computed blocks the Monitor tab's landing poll now
returns (all server-side so a curl client sees exactly what the page
draws, per the runbook's "curl is a complete client" architecture rule):

- server_state.py: a four-state discriminator (not_running | starting |
  running_no_metrics | running_with_metrics) that keys off an explicit API
  probe, never a metrics-scrape failure alone -- a connection-refused
  scrape used to be rendered as "no --enable-metrics", which is a
  different diagnosis than "not running at all". running_no_metrics is
  only ever reported with a passing API probe.
- rate_medians.py: medians behind the rate tiles, windowed over
  PROCESSING polls only -- idle polls never enter the window, so an idle
  server still reports the median of what it did while working, not a
  zero-diluted average.
- spill_tiers.py / tier_occupancy.py: one row per spill/offload tier type
  x place, with absent tiers PRINTED alongside their reason rather than
  hidden or shown as zero. Two tiers are deliberately never populated:
  the sgl-project#407 memtier registry (no production consumer, so reading it would
  print zeros that look measured) and the sgl-project#286 short-term register (no
  reachable byte ledger in this build) -- both documented as absent-by-
  construction, not omissions.

webui.py and live_metrics.py wire these into the existing landing poll;
metrics_reporter.py and metrics_collector.py get the small hooks the new
blocks read from.

Clean auto-merge on both touched docs (FEATURE_CATALOG.md and
rig-runbook.md) -- neither had a conflicting edit on this line since this
branch's base (3e3aca2, the sgl-project#510 merge). rig-runbook.md gains new
§8.0 documenting the shell-only verification recipe (curl + three python3
one-liners for state/medians/tiers) and the restart requirement: the
dashboard serves from one long-lived process, so a change is live only
after that process restarts -- done as the merge step below, not
mid-session. All 17 catalog sections and all 21 previously-preserved
additive paragraphs verified present (two-axis §1, matrix doctrine §2,
all six §12 families from sgl-project#510/sgl-project#513/sgl-project#511/sgl-project#487/qwen38-prep, sgl-project#514's five
families + value-pinning, sgl-project#499's resolution-ordering, sgl-project#517's two sgl-project#459
paragraphs, MERGE-DUTY owner-sites + SITREP §12, NVENC §13).

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The six directly touched/new test files: 517 passed, 0 failed.
- test/registered/unit/planner/ (full, HTSGLANG_TEST_MODEL_DIR set):
  2337 passed, 1 skipped, 352 subtests -- no failures.
- ruff check --select=F401,F821,UP037: 1 finding
  (live_metrics.py's LiveSnapshot import), confirmed pre-existing --
  already unused in the pre-merge base, only mentioned in a docstring,
  the line just shifted from sgl-project#522's unrelated insertions above it.
- codespell: 1 hit (webui.py:1786 "unparseable"), confirmed pre-existing
  at the identical line in the pre-merge base, in unrelated SVG
  chess-piece-parsing code this branch never touches.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 5, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
A live specimen (2026-08-06 21:08:18, pids 41283/41284/41285) was captured
with all three ranks in the SAME stack and held there for over eleven
minutes, byte-identical across two py-spy samples eleven minutes apart:

    Event.synchronize (torch/cuda/streams.py:254)
    _read_status_for_check (barlink_bar1.py:4527)
    check_aborted (barlink_bar1.py:4718)
    all_reduce -> attention_tensor_model_parallel_all_reduce
    _gather_hidden_states_and_residual (communicator.py:1287)

Not one abort line was emitted in that window, on any rank. Two independent
defects produced that silence, and this commit closes both.

1. THE GUARD WAITED ON THE FAULT. The staged status read issues its D2H on
   the COMPUTE stream, so the event that signals it is ordered behind the
   collective that just ran. The `max_lag` bound then forced
   `event.synchronize()` on that event -- a wait bounded by the collective,
   which is unbounded when the collective is what hung. The guard joined the
   wedge instead of reporting it. The wait now has a wall-clock deadline
   (SGLANG_BARLINK_BAR1_ABORT_SYNC_DEADLINE_MS, default 2000 ms, 0 restores
   the old blocking wait). Expiry leaves the read unresolved and the
   accumulated window intact, and logs the op and byte count -- the first
   host-visible symptom this path could previously emit at all.

2. THE WATCHDOG SILENTLY SKIPPED THIS TRANSPORT. `poll_status_words` walks
   registered transports and calls `poll_status_word` on each, looked up with
   `getattr(..., None)`. Only the DEVICE transport ever defined it.
   `BarlinkBar1Transport` did not, so the walk reached it, found nothing and
   moved on -- meaning the private-stream read, the one reader that still
   resolves while the compute stream is wedged, never ran on the BAR1 path.
   Ported from barlink_device.py, where the shape has been proven since sgl-project#517
   phase 2. The hot path now consults that sticky mirror first.

Escalation: an unbroken run of deadline expiries (default 30, i.e. ~60 s of a
stream that will not retire four bytes; a healthy round is ~46 ms) raises the
new Bar1CollectiveStalled, carrying group/rank/op/nbytes/expiries. It derives
from Bar1KernelAborted so existing handlers keep working, and it states
explicitly that the abort word is CLEAN -- a stall, not a trip. Before this,
the only actor that ever ended the wedge was an external supervisor, minutes
late and with no attribution.

WHAT THE SPECIMEN ALSO SHOWED, and what this commit does NOT yet fix. The
three ranks were stuck in collectives of DIFFERENT sizes: rank 0 at
nbytes=3420160 in decoder layer 6, ranks 1 and 2 at nbytes=12584960 in layer
5. The launch sampler agrees across the whole wedge window and adds a second
disagreeing pair (rank 0 at 4104192, ranks 1/2 at 6040768). A pre-wedge
control over 21:00-21:05 shows the ranks normally run IDENTICAL sizes
(10229760, 10301440, 10444800, 10670080, ... on all three), so the sizes are
not per-rank shards and the disagreement is abnormal. The two sizes ranks 1
and 2 froze on never occur anywhere in the preceding nine hours of their
logs. No Xid/NVRM/IOMMU fault, and the abort word stayed clean throughout, so
no kernel took its abort path: the collectives are waiting on peer
contributions that do not match. Root-causing that divergence needs the
collective history at the moment of the stall, which is exactly the artifact
the two defects above suppressed.

Tests (CUDA_VISIBLE_DEVICES=99, hermetic, no GPU):
  test/registered/unit/debug/            354 passed (327 before, 27 new)
  test/registered/unit/distributed/      unchanged; the 4 failures in
    test_uniform_decode_mem_603.py are pre-existing, reproduced with this
    patch reverted (_FakeScheduler has no attribute 'server_args').
  ruff check python/.../device_communicators/ + the new test: clean.

The new suite includes a falsifier that asserts the UNBOUNDED wait does not
return within 500 ms against an event that never completes, so the bounded
case is proven to be the difference rather than assumed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 6, 2026
…ion 18 reusable building blocks — 33 verified entries, 4 misdescriptions corrected (offload register is two modules; collective history lives in distributed/collective_census.py not debug_utils; byte-identity harness under tests/determinism; sgl-project#517 staged status-read is a pattern not a module), 13 previously undocumented modules incl. runtime_context (~400 consumers); merge-checklist rule; 66-test file:line consistency guard)
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 8, 2026
… probe

The SIGUSR1 launch dump must never touch device memory (its documented
no-sync constraint), so it could not print the ack barrier's round/
watermark words — and the on-card monotonicity probe had nothing to read.
The watchdog abort poll (sgl-project#517) already stages device words through a
private stream every 10 ms; it now also stages (round, mesh watermark,
a2a watermark) into a pinned mirror, and the dump prints that mirror.
Two spaced SIGUSR1 probes under replay load showing strict growth of all
three words per rank are the ack barrier's capture-safety proof through
the real replay path. 46 test items green (launch-diff + abort suites).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 16, 2026
…th collective

sgl-project#517 named five BAR1 broadcasts on the HOST path per NEXTN decode round.
Verified against the current tree, that inventory is now THREE, not five:
the three verify-result syncs were fused into one by pack_accept_payload
(#616c). The remaining two are the draft-pick sync in
_draft_extend_for_decode, which capture_safe_tp_broadcast still issues one
collective PER tensor from its loop.

This fuses that pair into one buffer. The picks have mixed dtypes
(topk_index integer, topk_p float32 at the SAME shape), so unlike the
int32 accept payload the fusion is byte-level. Contiguity is required
rather than coerced: reshape(-1) on a non-contiguous tensor returns a
copy, so an in-place unpack through it would silently drop the broadcast.

fuse is opt-in per call site and defaults OFF. _broadcast_draft_picks is
also called from inside the draft CUDA-graph capture (draft_forward), and
enabling fusion globally would change what a captured region allocates --
a capture change. Only the site that is both per-decode-round and outside
any captured region opts in; the prefill, catchup and in-loop sites stay
byte-identical.

Not claimed here: a ms/round figure. sgl-project#476's 6.64 pp Seam A cost was
measured with the pre-sgl-project#517 guard, which synchronised once per collective;
sgl-project#517 removed that sync, so the per-collective price must be re-measured on
metal rather than carried over. The A/B is window-gated.

Tests: test_draft_pick_fusion_524.py, 7 passed (red first: the three
symbols did not exist). Regression over test/registered/unit/debug
-k "616 or spec_util or draft_pick or eagle or broadcast": 351 passed,
4 failed. Those 4 are pre-existing and unrelated -- verified by reverting
both source files and re-running: test_barlink_bar1_abort_poll_616f.py
fails identically on the clean tree (SimpleNamespace missing _round_dev,
a sgl-project#624-class stub drift). ruff check clean on both changed files (the one
F401 in eagle_worker_v2.py is pre-existing on HEAD); ruff format applied.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…bort read BACK

The blocker named in bdbbaf4 is cleared, and the stop caller is wired.

THE BLINDNESS, restated because the fix is shaped by it. Since sgl-project#517 phase 2 the
watchdog is the only reader of a BarlinkDeviceTransport's abort word:
_arm_status_poll (barlink_device.py:1440) latches _abort_poll_active ONCE at
bring-up (:1478), the latch is one-way, check_aborted (:1540) short-circuits on
it, and _abort_code_seen is written only by poll_status_word (:1504) from the
watchdog thread. Stopping the watchdog therefore left the word unread by anyone
and check_aborted answered "not aborted" forever.

STEP 1 -- rearm_inline_reads() (barlink_abort_gate). Stopping the watchdog now
clears _abort_poll_active on every registered transport, so check_aborted falls
back to the pre-sgl-project#517 in-line device read. This is the degradation
should_poll_status already declares intended -- "the guard degrades to the
sgl-project#517-phase-1 behaviour rather than to blindness" -- which until now was true
only at bring-up, because that is where it is evaluated.

The re-arm runs BEFORE the join, so there is no window in which the reader is
gone while the latch still claims it reads. Pinned by a test that has the
worker record the latch at the moment it observes the stop event.

ABORT-WORD SEMANTICS UNTOUCHED: the word stays sticky, nothing writes it,
nothing clears a code. Only WHO READS IT changes, back to the reader it had
before sgl-project#517 phase 2. The in-line read it restores is the pre-sgl-project#517
per-collective device read, so its cost is paid only AFTER a stop -- during
teardown, never during serving.

SCOPE CORRECTION to my own bdbbaf4 wording ("every device transport"): the
blindness is the barlink_device family only. BarlinkBar1Transport.check_aborted
(barlink_bar1.py:5122) never consults the latch -- it always reads through
_read_status_for_check (:4889) -- so bar1 was never blind; there the latch only
short-circuits the watchdog's own poll. The re-arm covers both anyway, and a
test pins the asymmetry so a future edit cannot quietly make bar1 depend on it.

STEP 2 -- PeerWatchdog.stop, sibling shape. It used to do
`thread, self._thread = self._thread, None` BEFORE the join, so a timed-out
join left a live thread with no record and a second stop reported success for a
thread it had abandoned. Now: join first, clear the handle only on a real join,
keep it and log a loud WARNING on timeout, return
"joined"/"detached"/"already stopped".

DEADLINE 0.25 s, DERIVED NOT COPIED. _run waits poll_interval_s() (10 ms
default) between passes, so a cooperative exit lands within about one tick plus
one poll pass. 250 ms is ~25 ticks -- ample here, and 8x tighter than the 2 s
the kvso and lane siblings use, which wait on a host tier write and a CUDA
kernel launch respectively.

STEP 3 -- the caller. barlink_liveness.stop_watchdog() is the counterpart
ensure_watchdog never had. scheduler_teardown.release_barlink_watchdog() is
ALWAYS-STOP with no gate: the thread leaks on every boot with barlink liveness
on, whether or not the destroy is armed, so the destroy's flag must not leak
into it. Ordering is enforced BY CONSTRUCTION inside release_distributed, which
stops and joins the watchdog before it touches parallel_state; the scheduler
also calls it explicitly beforehand, but that call order is one careless edit
from inverting.

NOT TOUCHED, as instructed: destroy gating, barlink_comm close order, shm
unlink, abort-word semantics. No transport close path was edited.

Tests, hermetic (CUDA_VISIBLE_DEVICES=""), 24 new:
  test/registered/unit/distributed/test_barlink_watchdog_rearm_673.py
    RED-FIRST: 23 failed / 1 passed before the fix. The one that passed is the
      bar1 precision pin, which was already true -- it documents scope, it does
      not test the fix.
    the headline: an abort injected AFTER the stop is still SEEN
    latch cleared on every registered transport, before the thread is gone,
      heterogeneous registry left alone, a raising transport does not strand
      the others, idempotent, semantics unchanged on a clean word
    stop: timed-out join keeps the handle, detach is logged, a second stop does
      not claim success, clean join clears and is quiet, idempotent, safe when
      never started, deadline derived from the poll cadence
    wiring: graceful stops and re-arms, exception path leaves it alone,
      ALWAYS-STOP with no gate, no watchdog is a quiet no-op, never raises
    ordering: release_distributed records stop_watchdog BEFORE
      destroy_model_parallel; source-order and AST call-site pins

  can-fail by breakage, FOUR independent neuters:
    re-arm removed                          -> 5 red (incl. the blindness test)
    handle cleared before the join          -> 3 red
    ordering precondition removed           -> 1 red (the by-construction test)
    scheduler call site removed             -> 2 red

  regression, name-level with --color=no against a removed-diff baseline:
    distributed + managers + debug
      baseline 31 failed / 5199 passed
      with fix 29 failed / 5225 passed
    ZERO new failures. Two baseline failures did not reproduce
    (test_load_snapshot_backends ZmqRoundTrip, test_phase_flip_mover_streaming_631
    staging price) -- both unrelated to barlink teardown, so they are FLAKY, not
    fixed by this change, and no credit is claimed for them.

A harness bug of my own, found by running: the first watchdog stand-in used a
worker that returned immediately, so every stop reported "already stopped" and
four wiring tests were asserting nothing. The default worker now stays alive
until stopped.

FILING UPDATE: docs/FINDING_673_barlink_watchdog_stop_blocked.md carries the
resolution paragraph and the scope correction. The sgl-project#673 family now stands at
5 of 6 abort-shaped items desk-addressed, 1 filed with no action (the vendored
mooncake/mori/nixl workers -- no stop mechanism to wire, PD-only).

ON-METAL PROOF REMAINS DEFERRED for the whole family: the abort is intermittent
per process, so a single clean boot proves nothing. The fixes ride along on
future teardown-armed boots.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…c, cross-ref sgl-project#726

Docs only; no production file touched. Closes the three items my own retro-sweep
left open.

## 1. EVERY=32 (former section 19.3): NOT DEAD, but path-dependent

Traced the constant to its consumers rather than trusting either ledger entry.

  check_every() returns 1 when the env var is unset (barlink_abort_gate.py:
    189-197). The default is ONE, not 32 -- "=32" is only meaningful if
    somebody sets it.
  barlink_bar1.py:5177 reaches it UNCONDITIONALLY, after the
    abort_check_enabled / pending-launch gates. LIVE on bar1.
  barlink_device.py:1562 sits in the in-line fallback path BELOW sgl-project#517's
    _abort_poll_active early return, so on the device transport it is bypassed
    entirely whenever the watchdog feeds the flag.

BOTH LEDGER READINGS WERE PARTIALLY RIGHT AND NEITHER WAS COMPLETE. "Dead since
sgl-project#517" is true of the DEVICE path only -- that is exactly what phase 2 made
cheap -- while bar1 never got that treatment and still consults the knob.
NOTE_517 itself treated ..._EVERY as arm B3, expected to buy nothing on top of
the staged read; "buys nothing measurable" is not "the consumer was removed",
and that elision is how the two readings drifted apart. The sgl-project#431 utilisation
figure (72-84% -> 86-91%) predates sgl-project#517 and was never re-measured after it.

THE RECIPE QUESTION, answered: the production boot recipe does NOT set it --
absent from startkommandos-rig.md (grep -c = 0). So there is no dead knob in the
recipe misleading the next operator, which was the specific worry.

Worth recording: this is the SAME bar1-vs-device asymmetry I found independently
in sgl-project#673, where bar1's check_aborted never consults _abort_poll_active while the
device path short-circuits on it. Two investigations converged on one structural
fact about these two transports.

Section 19.3 is now empty and points at the settled 19.1 entry.

## 2. Stale spill-matrix doc (section 19.4): corrected at both sites

ANALYSE_spill_matrix_20260804.md claimed kvso x HiCache is mutually exclusive
and that "kvso cannot run on the production recipe at all". sgl-project#550 superseded
that: the combination is OPT-IN, gated on KVSO_ALLOW_HICACHE
(server_args.py:7502, `if os.environ.get("KVSO_ALLOW_HICACHE", "0") != "1":
raise`). Per the refusal's own text the two host pools are independent objects
with DISJOINT KEY SPACES and their pinned host RAM is summed by ONE JOINT BUDGET
GUARD instead of each validating alone; what remains is a measurement
(spill-copy vs prefetch contention), not a mechanism.

Fixed in section S1 and in the H15 matrix row, with the original text kept
inline so the change is legible rather than silently rewritten.

ANCHOR CORRECTION: the gate is at server_args.py:7502, NOT the :7385-7395 given
in the brief. Cited by symbol per the anchor rule -- the sgl-project#621 sweep found audit
line numbers had drifted, two of three into the wrong file.

## 3. sgl-project#726 cross-reference added to section 19.1

The INT8-KV IMMA-QK builder must CREATE the dtype surface, not find it: choices
at server_args.py:1026 carry no int8, the fp8 KV scale path is hard-coded
per-tensor, and the one per-group-scale precedent (MHATokenToKVPoolFP4)
dequantises eagerly before any backend sees it -- copying it buys VRAM savings
but NOT the bandwidth savings an int8-KV lane exists for. Budget the plumbing as
new work. The anchor is VERIFIED; the FP4-precedent detail is marked
UNVERIFIED-LEDGER (relayed from ANALYSE_726, not re-read here).

PRIOR-ART GATE honoured: grepped FEATURE_CATALOG, docs/dev and git log --all
--grep before each change; both absence claims carry the file:line of the
refusing gate (KVSO_ALLOW_HICACHE at :7502, kv_cache_dtype choices at :1026).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants