Skip to content

Fix Mistral model loading - #108

Merged
comaniac merged 1 commit into
mainfrom
cody/fix_mistral
Jan 26, 2024
Merged

comaniac merged 1 commit into
mainfrom
cody/fix_mistral

Conversation

@comaniac

Copy link
Copy Markdown
Contributor

Close #107

Co-authored with @johndun

@comaniac
comaniac merged commit cd68723 into main Jan 26, 2024
@comaniac
comaniac deleted the cody/fix_mistral branch January 26, 2024 17:38
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Co-authored-by: johndun <dunavent.jm@gmail.com>
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
* set default strides for outputs

* checkpoint

* Support input/output stride specification in search

* undo changes in cmake files
zhuyijie88 pushed a commit to zhuyijie88/sglang that referenced this pull request Sep 4, 2025
sammysun0711 pushed a commit to sammysun0711/sglang that referenced this pull request Dec 24, 2025
Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026
* Precision trace & Request dyeing
shadowxz109 pushed a commit to shadowxz109/sglang that referenced this pull request Mar 18, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…ifies four sites, --draft-kv-layout=dcp with named boot refusal until the draft-extend DCP split exists (sgl-project#108 slice 1)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…ify-shaped two-stage decomposition with prefix = seq_len - num_tokens_per_req (double-count falsifier), unconditional forces-prefix lockstep rule, separate graph-mode wrapper dict (flashinfer row latch), replicated default untouched (sgl-project#108 slice 2)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…he live hypothesis

Desk pass on the three open causes for the accept regression, plus the
regime-vs-cause correction written in so the next reader does not over-read the
discriminator the way the window did.

REGIME, NOT CAUSE (canonical framing). The k=2/k=4 pair localized where the
effect has HEADROOM to become visible -- at draft_tokens=2 acceptance was
already at its 2.0 ceiling, so a slightly worse draft had nowhere to show. It
said nothing about mechanism; every hypothesis below is compatible with it.

(1) COMPACTION vs DRAFT POOL ROW SPACE: CLEAN. build_dcp_weighted_kv_indices
reads global allocator ids out of req_to_token and compacts every one through
dcp_weighted_read_slots -- the exact inverse of the _dcp_masked_write packing.
Raw global ids never reach the pool. Draft and target SHARE req_to_token and
the allocator, and slice 1 sized the draft pool with the same
dcp_compact_pool_rows, so write and read land on the same row and every row is
in range. Checked the draft DECODE path too (slice 2 only built extend): its
DCP branch is gated on spec_info.kv_indptr is None, and nothing in
speculative/ ever assigns that field for the EAGLE draft inputs, so decode
takes the compaction branch as well. No bypass.

(2) RAGGED HEAD GEOMETRY: NOT APPLICABLE HERE. _replicated_kv_ragged_reindex
fires only under attn_kv_replicated(attn_tp_size, total_kv), i.e.
TP > num_kv_heads; this rig is 3 ranks over 8 kv heads, so the aligned split
gives whole GQA groups per rank and the reindex is correctly skipped. Its
docstring symptom is a near-match in shape, which is why it was checked, but
its trigger is absent. It becomes LIVE on TP > num_kv_heads hardware -- exactly
where this layout is worth enabling -- so it is re-checked there, not closed.

(3) LSE-MERGE DRIFT: the live hypothesis, and benign-shaped. The decisive
evidence was already in the window's artifacts and was not read at the time:
BOTH ARMS EMIT BYTE-IDENTICAL OUTPUT IDS, 64/64 on both prompts. The entire
difference is the verify count (16->19, 19->21). An indexing defect makes the
draft read wrong rows, which degrades predictions erratically and would not
leave the accepted trajectory untouched; a uniform handful of extra rounds at
identical output is the signature of the draft's logits differing in their last
bits, because the sharded prefix is read as per-rank partials and recombined by
LSE. That is the sgl-project#274/sgl-project#360 reassociation story one level down -- in the DRAFT
rather than in the verify.

WHAT IT MEANS FOR THE FEATURE. Correct, and on THIS rig not worth enabling:
VRAM byte-neutral (heads freely shardable, the two factors cancel) and accept
-10..-16% at identical output. The applicability rule is therefore sharper than
"it works": dcp pays only where the heads cannot be sharded further
(TP > num_kv_heads), which is where the VRAM win is the full shard factor. This
rig is the configuration where it costs and returns nothing -- a property of
the hardware, not a defect, and it belongs in the flag's help text.

RE-RUN PROPOSAL, one falsifier per hypothesis. H3: context-length sensitivity
-- drift keeps the RELATIVE accept gap roughly constant as context grows, a
latent indexing defect makes it GROW; run both arms at ~4k and compare against
the 64-token numbers in hand. H2: not falsifiable here, belongs to the
TP > num_kv_heads window. H1: no on-card falsifier; reopened only by H3's
gap-grows outcome. Recommended order: do the TP > num_kv_heads window FIRST --
it tests the feature where it is supposed to pay AND exercises H2 -- and treat
the context sweep as optional confirmation rather than a blocker.

Docs only; no code change, no cards touched.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…ope the TP>kv window

The durable outcome of the validation arc is a rule about WHERE this layout
pays, so it goes where an operator reads it.

FLAG HELP. --draft-kv-layout now states the measured trade rather than only the
mechanism: dcp saves draft VRAM only where the kv-heads cannot be sharded
further (TP > num_kv_heads), and there the saving is the full token-shard
factor ratio_r/S. On a head-shardable rig the token-shard and head-replication
factors CANCEL, so the layout is VRAM-byte-neutral while still costing
speculation efficiency -- measured on Qwen3.6-27B-FP8, TP=3 over 4 kv-heads,
vector [30,17,17]: output token ids byte-identical (64/64 on both probe
prompts) but verify rounds 16->19 and 19->21, accept -10..-16%. The text names
that cost as benign LSE reassociation in the DRAFT, not a correctness defect,
and tells the reader to leave the flag at 'replicated' on rigs whose heads
split evenly. Guidance text only: no gate changed, no auto-flip -- the choice
stays the user's.

One trap caught by the existing suite while writing it: a literal '%' in an
argparse help string makes --help raise for EVERY option, not just that one
(test_help_text_renders). Escaped as '%%'.

VEHICLE INVENTORY for the TP > num_kv_heads window, from a scan of the model
cache. The trigger is attn_kv_replicated(attn_tp_size, total_kv), i.e. strictly
TP > num_key_value_heads. TWO vehicles reach it at NATIVE TP=3, so the sgl-project#82
co-location axis is not needed at all:

  Qwen3.5-2B            2 kv /  8 q | 3.5 G | bf16 dense | MTP present
  Qwen3.6-35B-A3B-FP8   2 kv / 16 q |  31 G | FP8 MoE    | MTP present
  (also 2 kv: 35B-A3B GPTQ-Int4 20 G, AWQ-4bit 22 G; 122B-A10B needs sgl-project#77
   expert offload and is too heavy for this question)

The 27B family is 4 kv and would need TP=5 co-location -- deliberately not
used, one lever per window. Recommended pair: the 2B as the mechanism vehicle
(boots in seconds) and the 35B-A3B-FP8 as the realistic one.

PREDICTION TO TEST: under replication every rank holds all total_kv heads, so
the head factor is 1 on both sides and no longer cancels the token shard --
the saving should be the FULL shard factor, 47/27/27 % of the replicated draft
pool for a [30,17,17] vector.

WINDOW PROPOSAL: 2 questions, 4 boots ({2B, 35B-A3B-FP8} x {replicated, dcp}),
~25 min, standard reserve, no bar1, local CT999. Q1 does the VRAM win
materialize at the predicted factor (read the KV-allocated lines as before).
Q2 does accept hold and does H2 behave -- with two sub-checks that only exist
on this hardware: assert dcp_kv_replicated_heads is actually True (otherwise
the window measured nothing), and include a SHORT prompt in the probe set
because _replicated_kv_ragged_reindex is now on the live path and its
documented failure shape is corrupted short-prompt / first-chunk generation
while the gathered-q paged path stays correct. Out of scope: the H3 context
sweep (optional confirmation, not a blocker), throughput, TP=5.

Four expected outcomes are written out with what each would mean, including
the one that turns H2 from inapplicable into a real defect.

distributed + spec + server_args: zero failures unique to this branch, +446
passed. ruff/codespell/black clean. Docs and help text only; no cards touched.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
Prepared the approved window at the desk. Two pre-flight findings worth the
paper:

1. attn_kv_replicated is tp_plan_active(tp) and kv < tp -- it needs an
   INSTALLED plan. My first check called it bare, got False for every vehicle,
   and briefly looked like it had falsified the whole proposal. It had not; it
   had falsified the check. Consequence carried into the runsheet: the
   dcp_kv_replicated_heads precondition must be read off the RUNNING server,
   never computed at the desk.

2. Qwen3.5-2B at TP=3 with the rig vector [30,17,17] is 8q/2kv over [4,2,2] --
   verbatim the worked example in _replicated_kv_ragged_reindex's docstring
   ('rank0 held q0-3 -- all -> kv0 globally -- but gqa_local=2 sent q2-3 to
   kv1, corrupting short-prompt / first-chunk generation'). That makes the 2B
   the CANONICAL H2 vehicle, a better reason to pick it than its size.

Verified with the plan installed: 2B kv=2 q=8 -> replicated=True, q [4,2,2];
35B-A3B-FP8 kv=2 q=16 -> True, q [8,4,4]; 27B kv=4 -> False (the control).
Straddle check (sgl-project#105): the aligned units/groups keep every rank's q heads
inside one global kv head, so neither arm should hit the straddle ValueError
the <= threshold variant produced.

Runsheet at /spinning/gpu-battery-results/2026-08-02_108-tpgtkv/RUNSHEET.md:
4 arms, Q1 the predicted 47/27/27% shard factor, Q2 accept + the H2 short-prompt
probe with the precondition asserted first, ~25 min, standard reserve, no bar1,
kill by pid/pgid only. Docs only, no cards touched.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…the DEGRADED layout

The TP>num_kv_heads window (4 arms, ~20 min) confirmed Q1 to the digit and
inverted Q2, so the flag text now states both sides of the threshold with the
evidence for each.

Q1: the draft pool shards exactly at the predicted factor -- 655520 rows on
every rank (replicated) -> 307290/174131/174131 (dcp), K 0.31 -> 0.15/0.08/0.08
GB, total draft KV 1.86 -> 0.62 GB = -67%, which is exactly
1-(30+17+17)/(64*3). Identical on both vehicles, as expected: the draft pool
follows the vector, not the model size.

Q2, the discovery: ABOVE the threshold 'replicated' is not the cheaper setting,
it is the BROKEN one. At kv < tp the heads cannot be split, so a draft on the
replicated layout is asked for a head shard that does not exist -- the verified
output stays correct (the target governs it; output ids identical on every
graded pair) but its proposals are almost never accepted. Qwen3.5-2B: 61 verify
rounds for 64 tokens, accept 1.05, i.e. speculation paying full cost for
nothing, against 20 rounds / accept 3.20 on dcp. 35B-A3B the same shape, 32 ->
20. chain_quality_gate GREEN full marks on both vehicles.

So the help text no longer says 'dcp pays here'; it says which setting is
correct on each side of one threshold, and why:
  TP >  num_kv_heads -> dcp; replicated is DEGRADED (the 1.05-accept evidence)
  TP <= num_kv_heads -> replicated; dcp is VRAM-neutral and costs 10-16% accept

H2 (_replicated_kv_ragged_reindex) does not fire: the path is live per the
geometry log and short determined prompts generate correctly first-chunk.

PROBE-DESIGN RULE, folded into the doc and the runsheet because it generalises
beyond this task: a short probe used as a CORRUPTION check must have a
DETERMINED answer. The runsheet's original 'Hi.' probe was wrong -- at
temperature 0 an underdetermined prompt is rerouted entirely by a single
last-bit flip at position 0, so both arms stay fluent and the probe can only
ever report 'different', never 'wrong'. That is the instrument sgl-project#360/sgl-project#365 ruled
out, rebuilt by accident. Corrected form: mechanical sequences
(a b c d e f -> g h i j k l ...) or single-answer facts. Two further runsheet
corrections recorded: the sgl-project#328 gate may live on a newer commit than a feature
branch's base (run it from a tree that has it), and helpers are killed by
recorded pid, never by pkill -f.

distributed + spec + server_args: zero failures unique to this branch.
ruff/codespell/black clean. Help text and docs only.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…btrahend reds 45 assertions), falsified padding hypothesis documented, sharpened --draft-kv-layout guidance — above TP>num_kv_heads replicated is the DEGRADED layout (accept 1.05 vs 3.20), below it dcp costs 10-16% accept; determined-answer probe rule in the runsheet canon (sgl-project#108 s3)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…inst FEATURE_CATALOG.md

Adds docs/dev/AUDIT_500_mechanism_reach.md (239 catalog condition claims
checked against the code predicate that gates them: 39 WIDER, 63 NARROWER,
126 EXACT, 9 NOT-FOUND, 20 consolidated bug candidates in §5).

Corrects FEATURE_CATALOG.md §1 in place with file:line-cited predicates,
resolving real conflicts against the two intervening merges that also
touched §1 (sgl-project#492 attention/KV replication axis, sgl-project#496 merge-duty backlog):
- Uneven-DCP / Draft-KV-DCP bullet: replaced with the audit's precise
  predicates (dcp_size > 1 and get_tp_partition_ratios() is not None,
  distributed/utils.py:346; three entry paths; REPLICATED-KV geometry at
  utils.py:1048; MLA no-combine at dcp/comm.py:81) and the audit's S1-34
  NOT-FOUND finding that demotes "replicated is DEGRADED above TP>kv_heads"
  from a two-sided code rule to CLI-help prose only, surfacing the sgl-project#500-B3/
  sgl-project#108 open defect (--rank-kv-ratio route refused by the draft-KV-DCP gate).
  The sgl-project#492 "TWO DISTRIBUTION AXES" paragraph is kept unchanged immediately
  after it (additive, not contradicted by the audit).
- Matrix Doctrine (§2) joint-prefill paragraph: replaced the sgl-project#492
  correction's enumerator claim with the audit's more precise one
  (uneven_perf.py:4133-4148, placement.py:813, scheduler.py:5860, :6408,
  :5218) and folded the sgl-project#492 measured bracket/loose-ctx findings back in as
  "still the right empirical read on this rig, refined by sgl-project#500-B1" rather
  than dropping them.

Reconciles NOTE_485_joint_phase_vectors.md's sgl-project#492 in-place REFUTED markers
with the audit's own correction of the same passages (header blockquote,
"attention grid" table row, conclusion item 2): both corrections are kept,
in chronological order, with the audit framed as refining sgl-project#492 rather than
replacing it — no duplicate REFUTED markers, full correction history intact.

Verified: branch's own diff (3b7569f..1d64fe3) touches only the three
doc files, no *.py; all 17 FEATURE_CATALOG.md §§ present; the five additive
paragraphs from intervening merges (two-axis §1, matrix doctrine §2, sgl-project#491
DSpark §4, merge-duty §12, NVENC §13) all present; codespell clean on the
three touched files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… pool -- established unreachable, not fixed

sgl-project#487 audits whether the stock even-DCP allocator branch
(model_runner_kv_cache_mixin.py, the else of the allocator chain -- which
inflates both index space and page granularity by dcp_size, assuming a
token-sharded pool) can ever run against a draft worker at
--draft-kv-layout replicated (the default), whose pool sizing has the
opposite geometry (draft_pool_is_replicated) and which the allocator
selection never mentions. sgl-project#108 never audited this crossing.

Answer established WITHOUT a boot, by enumerating every producer of
is_draft_worker=True rather than assuming there is one shape: UNREACHABLE
on CUDA. Given dcp_size > 1 the stock branch is taken exactly when
rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative
draft worker cannot exist in that shape because
ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA
unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a
sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT
speculative, so leg 1 misses it -- closed instead by
_lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only
because "a draft pool implies a speculative algorithm" turned out false
once the producer set was enumerated, not assumed -- the new falsifier
pins that set so a third producer lands as a red test instead of a wrong
address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run
(is_hip() short-circuits before the CUDA branch), so the crossing IS
admitted there -- left alone, since this fork does not serve ROCm and a
desk-guessed change to address computation is exactly the sgl-project#345
right-token/wrong-slot class waiting to happen.

Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and
its script updates (scripts/dev/394_s2_proof/).

FEATURE_CATALOG.md §12 conflict as expected (this branch and the
already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the
same point, right after the §12 header): kept all six additive paragraphs
in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513,
Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512,
Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17
sections and every other previously-preserved additive paragraph
(two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's
resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13)
verified present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- test_stock_dcp_allocator_reach_487.py alone: 20 passed.
- test/registered/unit/distributed/ (full, since
  model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed,
  8 skipped, 723 subtests -- same 24 failing test IDs as the established
  baseline, byte-identical; the +20/+2 deltas are exactly this merge's own
  new test file.
- ruff check --select=F401,F821,UP037: clean.
- codespell: clean on all 8 touched/added files.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…project#636)

The handover onto a token-sharded decode pool is supported on exactly one
combination: page_size == 1, hisparse off, the mooncake transport, on the
uneven-TP replicated-KV layout. All four were already enforced -- and every
one of them only at RUNTIME. Three fire in disaggregation/decode.py
(:1109-1133) on the first send_metadata; the transport one fires from inside
the SENDER (nixl/conn.py:2511, mori/conn.py:1699), later still. So the server
boots healthy, reports 200, accepts traffic, and dies on the FIRST handover,
after the full weight load and in production.

That lateness is the defect, not the number of files: four loud failures are
fine, four loud failures that all arrive after the weight load are not. Every
input is decidable from ServerArgs, so the combination is decided at parse
time, reporting ALL violations at once rather than one reboot at a time.

PLACEMENT -- a correction to the recon, and the load-bearing part of this
commit. The suggested home was inside handle_pd_disaggregation, on the
grounds that all four flags are in scope there. Three are; dcp_size is not.
It is auto-set to tp_size inside _handle_uneven_tp, which __post_init__ calls
at :5957, AFTER _handle_pd_disaggregation at :5916 -- both verified in the
same function body. A gate in the hook would read the unresolved dcp_size,
conclude the token-shard path is not in force, and pass silently on exactly
the configuration it guards. That is worse than no gate, because it reads as
coverage.

The fork has been here before: sgl-project#108's _reject_unsupported_draft_kv_dcp sits
late for the same reason and records a MEASURED false refusal from reading
these inputs early ("rank_tp_ratio=auto-performance, dcp_size=1" rejecting a
correct boot). This gate follows that precedent and runs immediately after it,
which is also after the hook's own mooncake_tcp -> mooncake rewrite, so the
transport check sees the resolved name.

Inert unless it applies: returns early for a monolithic server, and for any
layout that is not dcp_size > 1 == tp_size. The decode-side runtime checks
stay as the backstop for anything arriving by another route.

Tests: test_pd_dcp_token_shard_contract_636.py, 8 hermetic + 6 subtests,
green. One contract test over the combination (each precondition refuses and
names itself; several violations report together; inert off the layout and off
PD), plus a mechanism-reach class that reads __post_init__'s source and pins
the ORDERING -- gate present, after _handle_uneven_tp, after the PD hook.
Two can-fail arms executed: moving the call up next to the PD hook turns
test_gate_runs_after_uneven_tp_resolves_dcp_size red, and dropping the
page_size precondition turns the contract test and its subtest red. The first
is the trap the placement correction exists for.

Base-vs-branch on test/registered/unit/server_args/ + unit/disaggregation/
with the same command: 762 passed, 0 failed on branch, so no regression is
possible. ruff clean on both changed files (server_args.py carries 355
pre-existing ruff findings, identical on base -- verified against
HEAD:server_args.py); codespell clean.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
Desk work for the gated GPU window: the wiring written down before it is
built, and the criterion the wiring will be held to. No code changes.

The lift criterion is a checkable table, L1-L9, each with the instrument that
decides it. L6 is the one worth arguing for: acceptance rate on the PD pair
must sit within the same-boot noise floor of the monolithic arm. A draft KV
that is wrong in a way that does not crash shows up ONLY as lower acceptance,
which reads as "this rig is slow" rather than as a defect -- the same
silent-degradation class as #631a's original auto-disable. Lifting the refusal
on L5 (same accepted-token sequence) alone would reintroduce it one layer
down.

W0 was read-only, so it was executed now rather than left as a step for the
window, and it changes the recommendation.

The draft pool's head count is decided by ONE predicate, read by both the pool
sizing and the attention backend so they cannot disagree
(owner.py:102 draft_pool_is_replicated). Under --draft-kv-layout replicated,
the default and what production runs, the draft pool is HEAD-SHARDED, which is
uneven under --rank-tp-ratio -- #631a's stated reason is correct there. Under
--draft-kv-layout dcp the draft pool takes "the identical target-model
treatment": full replicated kv-heads, token-sharded by the same weighted owner
rule (model_runner_kv_cache_mixin.py:2783-2791).

The second case dissolves the problem rather than solving it. A draft pool
shaped exactly like the target pool crosses the PD boundary by exactly the
mechanism that already carries the target pool -- the extra-layers
registration on shared token indices (prefill.py:186, decode.py:439) plus the
owned_ordinals scatter the R6 verdict established as already wired. No
canonical shipment, no head reslicing, nothing new on the wire. And production
sits inside sgl-project#108's covered shape item for item: NEXTN, topk 1, one draft
layer, rank_tp_ratio installed, dcp_size 3 (server_args.py:7784-7799).

So the recommendation is now path B first -- require --draft-kv-layout dcp for
a PD pair that wants speculation -- reusing sgl-project#108's owner-rule kernels and the
existing scatter instead of adding a third transfer shape.

This partly supersedes the contract landed in 214433e, and saying so is
worth more than defending it. That contract is not wrong and is not deleted:
it remains the answer for the default replicated (head-sharded) layout, its
geometry gate and version discipline are layout-agnostic, and
local_head_window is already the recorded workaround for sgl-project#641. Retiring a
proven desk artifact on an unproven expectation is how the fallback goes
missing exactly when it is needed, so path B gets demonstrated on-card first.

Path B is not free either, and the document says what it still owes: that the
draft pool's DCP treatment is identical at the TRANSFER boundary and not
merely similar (the extra-layers registration assumes matching per-rank item
lengths, and sgl-project#345 is the standing example of a stride mismatch that corrupts
silently), that --draft-kv-layout dcp composes with disaggregation_mode at all
(no boot has combined them), and the measured acceptance cost.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 7, 2026
…t, yet

Desk answer to the owed question, so the GPU window confirms instead of
investigates. Documentation only.

COMPOSITION VERDICT: UNCONSIDERED. draft_kv_layout appears in server_args.py
exactly twice -- the argument definition (:1517) and the sgl-project#108 gate (:7799) --
and the sgl-project#108 gate contains no reference to disaggregation, prefill or decode.
pd_disaggregation_hook.py contains no reference to draft_kv_layout. Only two
files mention both concepts at all and in both the mention is incidental.
Nothing refuses the combination and nothing supports it.

WHAT BREAKS FIRST, from code rather than prediction. Three facts compose into
a silent corruption. (1) The transport is draft-blind: mooncake/conn.py has
ZERO occurrences of "draft", and the draft pool rides as ordinary extra layers
under the comment "The indices are always shared with a target model"
(prefill.py:186-195, decode.py:439-448). (2) The decode arm rewrites
kv_indices into compact owner-rule rows ONCE, and that single array addresses
every registered buffer, target and draft alike (decode.py:1119-1125). (3) The
draft pool's row space depends on the layout:
model_runner_kv_cache_mixin.py:2821-2823 returns the FULL global context for
the default 'replicated' ("keeps the early return ... byte-identical") and
falls through to dcp_compact_pool_rows only under 'dcp'.

So on a DCP decode arm with the default draft layout, compact row indices are
applied to a pool sized to the full global context -- two different coordinate
systems, every draft row at the wrong address, the offset growing with the
slot id. That is sgl-project#345's right-token/wrong-slot drift, silent rather than a
crash. It is latent today only because #631a nulls speculative_algorithm on a
PD arm, so no draft worker exists and the registration never runs.

This is the strongest argument for path B so far, and it is not about
efficiency: under --draft-kv-layout dcp the draft pool takes
dcp_compact_pool_rows, the target's own coordinate system, so the shared
indices are correct BY CONSTRUCTION. Path B is the only one of the two layouts
on which the existing shared-index transfer is correct at all.

Owed item (1) resolves with it: under 'dcp' the draft pool carries
get_total_num_kv_heads() (:2788-2791), matching the target's treatment under
uneven_dcp_kv_replicated, so per-rank item lengths agree by the same
construction.

NEW REFUSAL UNCOVERED, added to the criterion as L10 and L11: the two arms
must agree on --draft-kv-layout, and a DCP decode arm must be on 'dcp'.
Guard 1 cannot cover the first -- draft_kv_layout is deliberately not part of
compute_model_identity_hash, being a parallelism decision rather than a
weights one. Its natural home is beside the layout version in
DraftKvCanonicalLayout, which is a second reason that contract stays.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 25, 2026
…he flip (3 front switches, all default off)

MEASURED, boots weg2xsn429/430/432/433/435 (45 P-ladder requests, front log,
tool /root/.claude/jobs/1ab4cd30/tmp/flip_roundtrip_27b.py). The ~4 s the ladder
reads as "client wall minus P wall" (median 3.95-4.24 s) is a ROUND TRIP, not
one flip:

  a  route -> D->P flip begin        74-133 ms   controller tick (0.2 s poll)
  b  D->P flip (front)             1.56-1.60 s
  c  D->P done -> P leg 1 start     199-203 ms   controller tick, EVERY request
  -- P prefill (not counted) --
  f  P->D flip (front)             1.79-1.88 s
  g  P->D done -> D first token     215-275 ms   D readmission (eager extend)
  P end -> D first token (Flipzeit) 2.01-2.20 s

Inside b, between the kv wake answering and `done`, the residue reading
(`ps` + `nvidia-smi` as blocking subprocesses on the front's event loop) costs
44-48 ms (median per boot) after P is already awake; in f it costs 56-67 ms
but is off the first-token path there. None of the three touches f/g: the
dormant D admission already posts leg 2 during the P->D flip.

The switches (env, read once at front start, named in one WEG2-FLIPFAST line):

* SGLANG_WEG2_CTL_KICK_ARRIVAL=1 (F2): a request joining the P queue -- the
  four enqueue sites of handle_generate / leg2 reroute / W50 requeue -- and D's
  LAST leg 2 ending while the P queue holds work wake the controller at once.
* SGLANG_WEG2_CTL_KICK_AFTER_FLIP=1 (F3): a completed D->P flip wakes it at
  once, so P's drain starts right after the flip instead of one tick later.
* SGLANG_WEG2_DC_OFF_PATH=1 (F1): once nothing gates on the residue reading
  (W19 settled at the first D sleep with a non-empty reading, the group's
  dormant image sampled), it runs in the default executor after the flip
  closed; same WEG2-DC lines, the flip record's dc_mib is filled in place
  (dc_off_path=True marks it), plus one WEG2-DC-OFFPATH line. The first sleep
  of each group and every D sleep after an empty first reading stay
  synchronous.

A kick only ends ONE wait early; every latch (economics, dwell, fairness,
hand-off window) is evaluated exactly as on a tick. No kick may race
d_admitter: the after-flip kick follows only a D->P flip (after P->D nothing
waits on the controller), and an arrival kick is held while prefilled
requests still wait in _ready_for_d (the D->P decision does not read it).
Off = asyncio.sleep(0.2) and the inline reading, byte for byte. The literal
`self.queue.append` stays (test_weg2_no_route_1290 pins it); Fronts built
via __new__ in other suites read as all-off.

Expected on the metal with all three on: client-P -0.33 s per P-routed
request (a+c+dc: 0.086+0.203+0.046 s on xsn435), Flipzeit P-end -> first
token unchanged (2.0-2.2 s; its levers are the legs and the readmission).

Tests (desk, CUDA_VISIBLE_DEVICES="", via test27b.sh):
* test/registered/unit/weg2/test_weg2_flipfast_27b.py, 20 tests: red on
  9f0ac9c (15 failed / 5 passed -- the 5 are the default-path, W19 and
  source-pin guards that must pass there), green on this tree (20 passed).
* every suite that imports the front (58 files): 1342 passed; 32 failed, and
  all 32 fail identically on 9f0ac9c (known red: 4 launcher argv tests on
  a SimpleNamespace without weg2_weight_source, the flip_cost '"tags": family'
  pin, the xsn288 weight_updater pin, no_route_1290 end-to-end on a __new__
  Front without .vision, parked_abort_w1b "no current event loop", and 22 in
  sched_fix3/fix7/slice_a/rpc_conn/s3s4/text_only/w101/wake_kv_early/wcode/
  x_seed). 0 new failures.

NF line (H71, onto c701dd4 + H62): picked with -x, three conflicts in
weg2/front.py resolved line by line, nothing else changed.
- Module level: only the flipfast block (the three CTL_*/DC_OFF_PATH
  switches, CTL_TICK_S, CTL_KICK_REASONS, _env_switch_on) taken; its 27B
  neighbours _sleep_gate_by_name / _measured_record_stat_key (W100 sleep
  gate, not on NF and ordered off there) left out.
- flip(): the F1 if/else wraps NF's own inline residue reading, kept verbatim
  and re-indented under `else:` -- dormant_residue_over and the
  --weights-resident W19 exemption (fnFL2 v21) stay in the synchronous branch,
  which is the first D sleep. NF's sample_dormant_image samples once per group
  as on 27B, so _dc_reading_deferrable holds unchanged.
- Methods: the flipfast methods follow NF's _adopt_first_flip (sgl-project#108); the
  controller keeps `await self._adopt_first_flip()` before its loop, the
  loop's tick (the same `await asyncio.sleep(0.2)` as on 27B) is _ctl_wait().
H61's client_io / _STREAM_FINISH_RE are outside every hunk and unchanged.
The P-intake-stall requeue (`appendleft`, weg2xsn272) gets no kick, as on 27B.

(cherry picked from commit 502e97e)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 29, 2026
…ung -- Deckung ueber alle Tags + Riegel fuer ungedeckte Tensoren

Prior-Art statt zweiter Mechanik: D ohne Platte gibt es seit sgl-project#108/sgl-project#109
(22.09., --weg2-d-adopt on: D laedt --load-format dummy, W115 sperrt jeden
Forward, der Erstflip bringt Dense UND Experten-Praefix ueber die Legs --
die Platztausch-Puffer sgl-project#135 stehen in Manifest/Plan/Coverage --, kalte
Zeilen kommen aus dem Store sgl-project#109). Am Metall zuletzt w87 (22.09.): 0
Presplit-Zeilen auf D. Seitdem sammelt der Wake je TAG (sgl-project#1374), und genau
dort war die Naht offen:

* _weg2_wake_reload_weights kehrte im per-Tag-Zweig ZURUECK, bevor der
  sgl-project#108-Block lief -- mark_adopted wurde nie gerufen, W115 fiel nie.
* _weg2_last_inject_cover hielt nur den LETZTEN Tag (filled eines Tags
  gegen expected des ganzen Plans) -- selbst erreicht haette die Deckung
  nie gereicht.
* Die Deckung zaehlte nur Plan-Deskriptoren: ein Tensor ohne Deskriptor
  (Peer haelt ihn nicht) behielte still seine Dummy-Bytes, waehrend der
  Riegel faellt.

Fix: _weg2_adopt_accumulate sammelt je Tag (filled je Tag, expected =
ganzer Plan, geschriebene Namen ohne ZEROFILL), _weg2_adopt_settle laeuft
in BEIDEN Wake-Formen, _weg2_adopt_uncovered vergleicht jeden Tensor des
Rangs (_weg2_rank_param_table) mit den geschriebenen Namen;
SGLANG_WEG2_D_ADOPT_UNCOVERED=refuse (Default unter Adoption) laesst den
Riegel dann stehen, =log nennt nur. Ohne --weg2-d-adopt ist alles inert
(accumulate/settle tun nichts, solange keine Platzhalter stehen).

Marker: 'sgl-project#108 ADOPT-COVER filled=N expected=N uncovered=0 -> PLATZHALTER
GELOEST'; rot: 'RIEGEL BLEIBT', 'sgl-project#108 ADOPT-UNCOVERED n=.. first=..'.

Test test_adopt_per_tag_bootzeit3_0929 (7): auf 4e153b8 rot (API fehlt),
gruen mit Fix; Nachbarn adopt_erstboot/repack_adoption/store_als_quelle
gruen. weg2-Wake-Suiten (flip_legs_h11, xchg_* u. a.) sammeln am Desk
nicht ('Invalid device id', identisch auf der Basis).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 29, 2026
…ung -- Deckung ueber alle Tags + Riegel fuer ungedeckte Tensoren

Prior-Art statt zweiter Mechanik: D ohne Platte gibt es seit sgl-project#108/sgl-project#109
(22.09., --weg2-d-adopt on: D laedt --load-format dummy, W115 sperrt jeden
Forward, der Erstflip bringt Dense UND Experten-Praefix ueber die Legs --
die Platztausch-Puffer sgl-project#135 stehen in Manifest/Plan/Coverage --, kalte
Zeilen kommen aus dem Store sgl-project#109). Am Metall zuletzt w87 (22.09.): 0
Presplit-Zeilen auf D. Seitdem sammelt der Wake je TAG (sgl-project#1374), und genau
dort war die Naht offen:

* _weg2_wake_reload_weights kehrte im per-Tag-Zweig ZURUECK, bevor der
  sgl-project#108-Block lief -- mark_adopted wurde nie gerufen, W115 fiel nie.
* _weg2_last_inject_cover hielt nur den LETZTEN Tag (filled eines Tags
  gegen expected des ganzen Plans) -- selbst erreicht haette die Deckung
  nie gereicht.
* Die Deckung zaehlte nur Plan-Deskriptoren: ein Tensor ohne Deskriptor
  (Peer haelt ihn nicht) behielte still seine Dummy-Bytes, waehrend der
  Riegel faellt.

Fix: _weg2_adopt_accumulate sammelt je Tag (filled je Tag, expected =
ganzer Plan, geschriebene Namen ohne ZEROFILL), _weg2_adopt_settle laeuft
in BEIDEN Wake-Formen, _weg2_adopt_uncovered vergleicht jeden Tensor des
Rangs (_weg2_rank_param_table) mit den geschriebenen Namen;
SGLANG_WEG2_D_ADOPT_UNCOVERED=refuse (Default unter Adoption) laesst den
Riegel dann stehen, =log nennt nur. Ohne --weg2-d-adopt ist alles inert
(accumulate/settle tun nichts, solange keine Platzhalter stehen).

Marker: 'sgl-project#108 ADOPT-COVER filled=N expected=N uncovered=0 -> PLATZHALTER
GELOEST'; rot: 'RIEGEL BLEIBT', 'sgl-project#108 ADOPT-UNCOVERED n=.. first=..'.

Test test_adopt_per_tag_bootzeit3_0929 (7): auf 4e153b8 rot (API fehlt),
gruen mit Fix; Nachbarn adopt_erstboot/repack_adoption/store_als_quelle
gruen. weg2-Wake-Suiten (flip_legs_h11, xchg_* u. a.) sammeln am Desk
nicht ('Invalid device id', identisch auf der Basis).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mistral model no longer loads following PR#101

1 participant