Repository navigation
Fix Mistral model loading - #108
Merged
Merged
Conversation
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Co-authored-by: johndun <dunavent.jm@gmail.com>
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
* set default strides for outputs * checkpoint * Support input/output stride specification in search * undo changes in cmake files
zhuyijie88
pushed a commit
to zhuyijie88/sglang
that referenced
this pull request
Sep 4, 2025
sammysun0711
pushed a commit
to sammysun0711/sglang
that referenced
this pull request
Dec 24, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
* Precision trace & Request dyeing
shadowxz109
pushed a commit
to shadowxz109/sglang
that referenced
this pull request
Mar 18, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 31, 2026
…ifies four sites, --draft-kv-layout=dcp with named boot refusal until the draft-extend DCP split exists (sgl-project#108 slice 1)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…ify-shaped two-stage decomposition with prefix = seq_len - num_tokens_per_req (double-count falsifier), unconditional forces-prefix lockstep rule, separate graph-mode wrapper dict (flashinfer row latch), replicated default untouched (sgl-project#108 slice 2)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…he live hypothesis Desk pass on the three open causes for the accept regression, plus the regime-vs-cause correction written in so the next reader does not over-read the discriminator the way the window did. REGIME, NOT CAUSE (canonical framing). The k=2/k=4 pair localized where the effect has HEADROOM to become visible -- at draft_tokens=2 acceptance was already at its 2.0 ceiling, so a slightly worse draft had nowhere to show. It said nothing about mechanism; every hypothesis below is compatible with it. (1) COMPACTION vs DRAFT POOL ROW SPACE: CLEAN. build_dcp_weighted_kv_indices reads global allocator ids out of req_to_token and compacts every one through dcp_weighted_read_slots -- the exact inverse of the _dcp_masked_write packing. Raw global ids never reach the pool. Draft and target SHARE req_to_token and the allocator, and slice 1 sized the draft pool with the same dcp_compact_pool_rows, so write and read land on the same row and every row is in range. Checked the draft DECODE path too (slice 2 only built extend): its DCP branch is gated on spec_info.kv_indptr is None, and nothing in speculative/ ever assigns that field for the EAGLE draft inputs, so decode takes the compaction branch as well. No bypass. (2) RAGGED HEAD GEOMETRY: NOT APPLICABLE HERE. _replicated_kv_ragged_reindex fires only under attn_kv_replicated(attn_tp_size, total_kv), i.e. TP > num_kv_heads; this rig is 3 ranks over 8 kv heads, so the aligned split gives whole GQA groups per rank and the reindex is correctly skipped. Its docstring symptom is a near-match in shape, which is why it was checked, but its trigger is absent. It becomes LIVE on TP > num_kv_heads hardware -- exactly where this layout is worth enabling -- so it is re-checked there, not closed. (3) LSE-MERGE DRIFT: the live hypothesis, and benign-shaped. The decisive evidence was already in the window's artifacts and was not read at the time: BOTH ARMS EMIT BYTE-IDENTICAL OUTPUT IDS, 64/64 on both prompts. The entire difference is the verify count (16->19, 19->21). An indexing defect makes the draft read wrong rows, which degrades predictions erratically and would not leave the accepted trajectory untouched; a uniform handful of extra rounds at identical output is the signature of the draft's logits differing in their last bits, because the sharded prefix is read as per-rank partials and recombined by LSE. That is the sgl-project#274/sgl-project#360 reassociation story one level down -- in the DRAFT rather than in the verify. WHAT IT MEANS FOR THE FEATURE. Correct, and on THIS rig not worth enabling: VRAM byte-neutral (heads freely shardable, the two factors cancel) and accept -10..-16% at identical output. The applicability rule is therefore sharper than "it works": dcp pays only where the heads cannot be sharded further (TP > num_kv_heads), which is where the VRAM win is the full shard factor. This rig is the configuration where it costs and returns nothing -- a property of the hardware, not a defect, and it belongs in the flag's help text. RE-RUN PROPOSAL, one falsifier per hypothesis. H3: context-length sensitivity -- drift keeps the RELATIVE accept gap roughly constant as context grows, a latent indexing defect makes it GROW; run both arms at ~4k and compare against the 64-token numbers in hand. H2: not falsifiable here, belongs to the TP > num_kv_heads window. H1: no on-card falsifier; reopened only by H3's gap-grows outcome. Recommended order: do the TP > num_kv_heads window FIRST -- it tests the feature where it is supposed to pay AND exercises H2 -- and treat the context sweep as optional confirmation rather than a blocker. Docs only; no code change, no cards touched.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…ope the TP>kv window The durable outcome of the validation arc is a rule about WHERE this layout pays, so it goes where an operator reads it. FLAG HELP. --draft-kv-layout now states the measured trade rather than only the mechanism: dcp saves draft VRAM only where the kv-heads cannot be sharded further (TP > num_kv_heads), and there the saving is the full token-shard factor ratio_r/S. On a head-shardable rig the token-shard and head-replication factors CANCEL, so the layout is VRAM-byte-neutral while still costing speculation efficiency -- measured on Qwen3.6-27B-FP8, TP=3 over 4 kv-heads, vector [30,17,17]: output token ids byte-identical (64/64 on both probe prompts) but verify rounds 16->19 and 19->21, accept -10..-16%. The text names that cost as benign LSE reassociation in the DRAFT, not a correctness defect, and tells the reader to leave the flag at 'replicated' on rigs whose heads split evenly. Guidance text only: no gate changed, no auto-flip -- the choice stays the user's. One trap caught by the existing suite while writing it: a literal '%' in an argparse help string makes --help raise for EVERY option, not just that one (test_help_text_renders). Escaped as '%%'. VEHICLE INVENTORY for the TP > num_kv_heads window, from a scan of the model cache. The trigger is attn_kv_replicated(attn_tp_size, total_kv), i.e. strictly TP > num_key_value_heads. TWO vehicles reach it at NATIVE TP=3, so the sgl-project#82 co-location axis is not needed at all: Qwen3.5-2B 2 kv / 8 q | 3.5 G | bf16 dense | MTP present Qwen3.6-35B-A3B-FP8 2 kv / 16 q | 31 G | FP8 MoE | MTP present (also 2 kv: 35B-A3B GPTQ-Int4 20 G, AWQ-4bit 22 G; 122B-A10B needs sgl-project#77 expert offload and is too heavy for this question) The 27B family is 4 kv and would need TP=5 co-location -- deliberately not used, one lever per window. Recommended pair: the 2B as the mechanism vehicle (boots in seconds) and the 35B-A3B-FP8 as the realistic one. PREDICTION TO TEST: under replication every rank holds all total_kv heads, so the head factor is 1 on both sides and no longer cancels the token shard -- the saving should be the FULL shard factor, 47/27/27 % of the replicated draft pool for a [30,17,17] vector. WINDOW PROPOSAL: 2 questions, 4 boots ({2B, 35B-A3B-FP8} x {replicated, dcp}), ~25 min, standard reserve, no bar1, local CT999. Q1 does the VRAM win materialize at the predicted factor (read the KV-allocated lines as before). Q2 does accept hold and does H2 behave -- with two sub-checks that only exist on this hardware: assert dcp_kv_replicated_heads is actually True (otherwise the window measured nothing), and include a SHORT prompt in the probe set because _replicated_kv_ragged_reindex is now on the live path and its documented failure shape is corrupted short-prompt / first-chunk generation while the gathered-q paged path stays correct. Out of scope: the H3 context sweep (optional confirmation, not a blocker), throughput, TP=5. Four expected outcomes are written out with what each would mean, including the one that turns H2 from inapplicable into a real defect. distributed + spec + server_args: zero failures unique to this branch, +446 passed. ruff/codespell/black clean. Docs and help text only; no cards touched.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
Prepared the approved window at the desk. Two pre-flight findings worth the
paper:
1. attn_kv_replicated is tp_plan_active(tp) and kv < tp -- it needs an
INSTALLED plan. My first check called it bare, got False for every vehicle,
and briefly looked like it had falsified the whole proposal. It had not; it
had falsified the check. Consequence carried into the runsheet: the
dcp_kv_replicated_heads precondition must be read off the RUNNING server,
never computed at the desk.
2. Qwen3.5-2B at TP=3 with the rig vector [30,17,17] is 8q/2kv over [4,2,2] --
verbatim the worked example in _replicated_kv_ragged_reindex's docstring
('rank0 held q0-3 -- all -> kv0 globally -- but gqa_local=2 sent q2-3 to
kv1, corrupting short-prompt / first-chunk generation'). That makes the 2B
the CANONICAL H2 vehicle, a better reason to pick it than its size.
Verified with the plan installed: 2B kv=2 q=8 -> replicated=True, q [4,2,2];
35B-A3B-FP8 kv=2 q=16 -> True, q [8,4,4]; 27B kv=4 -> False (the control).
Straddle check (sgl-project#105): the aligned units/groups keep every rank's q heads
inside one global kv head, so neither arm should hit the straddle ValueError
the <= threshold variant produced.
Runsheet at /spinning/gpu-battery-results/2026-08-02_108-tpgtkv/RUNSHEET.md:
4 arms, Q1 the predicted 47/27/27% shard factor, Q2 accept + the H2 short-prompt
probe with the precondition asserted first, ~25 min, standard reserve, no bar1,
kill by pid/pgid only. Docs only, no cards touched.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…the DEGRADED layout The TP>num_kv_heads window (4 arms, ~20 min) confirmed Q1 to the digit and inverted Q2, so the flag text now states both sides of the threshold with the evidence for each. Q1: the draft pool shards exactly at the predicted factor -- 655520 rows on every rank (replicated) -> 307290/174131/174131 (dcp), K 0.31 -> 0.15/0.08/0.08 GB, total draft KV 1.86 -> 0.62 GB = -67%, which is exactly 1-(30+17+17)/(64*3). Identical on both vehicles, as expected: the draft pool follows the vector, not the model size. Q2, the discovery: ABOVE the threshold 'replicated' is not the cheaper setting, it is the BROKEN one. At kv < tp the heads cannot be split, so a draft on the replicated layout is asked for a head shard that does not exist -- the verified output stays correct (the target governs it; output ids identical on every graded pair) but its proposals are almost never accepted. Qwen3.5-2B: 61 verify rounds for 64 tokens, accept 1.05, i.e. speculation paying full cost for nothing, against 20 rounds / accept 3.20 on dcp. 35B-A3B the same shape, 32 -> 20. chain_quality_gate GREEN full marks on both vehicles. So the help text no longer says 'dcp pays here'; it says which setting is correct on each side of one threshold, and why: TP > num_kv_heads -> dcp; replicated is DEGRADED (the 1.05-accept evidence) TP <= num_kv_heads -> replicated; dcp is VRAM-neutral and costs 10-16% accept H2 (_replicated_kv_ragged_reindex) does not fire: the path is live per the geometry log and short determined prompts generate correctly first-chunk. PROBE-DESIGN RULE, folded into the doc and the runsheet because it generalises beyond this task: a short probe used as a CORRUPTION check must have a DETERMINED answer. The runsheet's original 'Hi.' probe was wrong -- at temperature 0 an underdetermined prompt is rerouted entirely by a single last-bit flip at position 0, so both arms stay fluent and the probe can only ever report 'different', never 'wrong'. That is the instrument sgl-project#360/sgl-project#365 ruled out, rebuilt by accident. Corrected form: mechanical sequences (a b c d e f -> g h i j k l ...) or single-answer facts. Two further runsheet corrections recorded: the sgl-project#328 gate may live on a newer commit than a feature branch's base (run it from a tree that has it), and helpers are killed by recorded pid, never by pkill -f. distributed + spec + server_args: zero failures unique to this branch. ruff/codespell/black clean. Help text and docs only.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…btrahend reds 45 assertions), falsified padding hypothesis documented, sharpened --draft-kv-layout guidance — above TP>num_kv_heads replicated is the DEGRADED layout (accept 1.05 vs 3.20), below it dcp costs 10-16% accept; determined-answer probe rule in the runsheet canon (sgl-project#108 s3)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…inst FEATURE_CATALOG.md Adds docs/dev/AUDIT_500_mechanism_reach.md (239 catalog condition claims checked against the code predicate that gates them: 39 WIDER, 63 NARROWER, 126 EXACT, 9 NOT-FOUND, 20 consolidated bug candidates in §5). Corrects FEATURE_CATALOG.md §1 in place with file:line-cited predicates, resolving real conflicts against the two intervening merges that also touched §1 (sgl-project#492 attention/KV replication axis, sgl-project#496 merge-duty backlog): - Uneven-DCP / Draft-KV-DCP bullet: replaced with the audit's precise predicates (dcp_size > 1 and get_tp_partition_ratios() is not None, distributed/utils.py:346; three entry paths; REPLICATED-KV geometry at utils.py:1048; MLA no-combine at dcp/comm.py:81) and the audit's S1-34 NOT-FOUND finding that demotes "replicated is DEGRADED above TP>kv_heads" from a two-sided code rule to CLI-help prose only, surfacing the sgl-project#500-B3/ sgl-project#108 open defect (--rank-kv-ratio route refused by the draft-KV-DCP gate). The sgl-project#492 "TWO DISTRIBUTION AXES" paragraph is kept unchanged immediately after it (additive, not contradicted by the audit). - Matrix Doctrine (§2) joint-prefill paragraph: replaced the sgl-project#492 correction's enumerator claim with the audit's more precise one (uneven_perf.py:4133-4148, placement.py:813, scheduler.py:5860, :6408, :5218) and folded the sgl-project#492 measured bracket/loose-ctx findings back in as "still the right empirical read on this rig, refined by sgl-project#500-B1" rather than dropping them. Reconciles NOTE_485_joint_phase_vectors.md's sgl-project#492 in-place REFUTED markers with the audit's own correction of the same passages (header blockquote, "attention grid" table row, conclusion item 2): both corrections are kept, in chronological order, with the audit framed as refining sgl-project#492 rather than replacing it — no duplicate REFUTED markers, full correction history intact. Verified: branch's own diff (3b7569f..1d64fe3) touches only the three doc files, no *.py; all 17 FEATURE_CATALOG.md §§ present; the five additive paragraphs from intervening merges (two-axis §1, matrix doctrine §2, sgl-project#491 DSpark §4, merge-duty §12, NVENC §13) all present; codespell clean on the three touched files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… pool -- established unreachable, not fixed sgl-project#487 audits whether the stock even-DCP allocator branch (model_runner_kv_cache_mixin.py, the else of the allocator chain -- which inflates both index space and page granularity by dcp_size, assuming a token-sharded pool) can ever run against a draft worker at --draft-kv-layout replicated (the default), whose pool sizing has the opposite geometry (draft_pool_is_replicated) and which the allocator selection never mentions. sgl-project#108 never audited this crossing. Answer established WITHOUT a boot, by enumerating every producer of is_draft_worker=True rather than assuming there is one shape: UNREACHABLE on CUDA. Given dcp_size > 1 the stock branch is taken exactly when rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative draft worker cannot exist in that shape because ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT speculative, so leg 1 misses it -- closed instead by _lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only because "a draft pool implies a speculative algorithm" turned out false once the producer set was enumerated, not assumed -- the new falsifier pins that set so a third producer lands as a red test instead of a wrong address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run (is_hip() short-circuits before the CUDA branch), so the crossing IS admitted there -- left alone, since this fork does not serve ROCm and a desk-guessed change to address computation is exactly the sgl-project#345 right-token/wrong-slot class waiting to happen. Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and its script updates (scripts/dev/394_s2_proof/). FEATURE_CATALOG.md §12 conflict as expected (this branch and the already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the same point, right after the §12 header): kept all six additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17 sections and every other previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_stock_dcp_allocator_reach_487.py alone: 20 passed. - test/registered/unit/distributed/ (full, since model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed, 8 skipped, 723 subtests -- same 24 failing test IDs as the established baseline, byte-identical; the +20/+2 deltas are exactly this merge's own new test file. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 8 touched/added files.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
…project#636) The handover onto a token-sharded decode pool is supported on exactly one combination: page_size == 1, hisparse off, the mooncake transport, on the uneven-TP replicated-KV layout. All four were already enforced -- and every one of them only at RUNTIME. Three fire in disaggregation/decode.py (:1109-1133) on the first send_metadata; the transport one fires from inside the SENDER (nixl/conn.py:2511, mori/conn.py:1699), later still. So the server boots healthy, reports 200, accepts traffic, and dies on the FIRST handover, after the full weight load and in production. That lateness is the defect, not the number of files: four loud failures are fine, four loud failures that all arrive after the weight load are not. Every input is decidable from ServerArgs, so the combination is decided at parse time, reporting ALL violations at once rather than one reboot at a time. PLACEMENT -- a correction to the recon, and the load-bearing part of this commit. The suggested home was inside handle_pd_disaggregation, on the grounds that all four flags are in scope there. Three are; dcp_size is not. It is auto-set to tp_size inside _handle_uneven_tp, which __post_init__ calls at :5957, AFTER _handle_pd_disaggregation at :5916 -- both verified in the same function body. A gate in the hook would read the unresolved dcp_size, conclude the token-shard path is not in force, and pass silently on exactly the configuration it guards. That is worse than no gate, because it reads as coverage. The fork has been here before: sgl-project#108's _reject_unsupported_draft_kv_dcp sits late for the same reason and records a MEASURED false refusal from reading these inputs early ("rank_tp_ratio=auto-performance, dcp_size=1" rejecting a correct boot). This gate follows that precedent and runs immediately after it, which is also after the hook's own mooncake_tcp -> mooncake rewrite, so the transport check sees the resolved name. Inert unless it applies: returns early for a monolithic server, and for any layout that is not dcp_size > 1 == tp_size. The decode-side runtime checks stay as the backstop for anything arriving by another route. Tests: test_pd_dcp_token_shard_contract_636.py, 8 hermetic + 6 subtests, green. One contract test over the combination (each precondition refuses and names itself; several violations report together; inert off the layout and off PD), plus a mechanism-reach class that reads __post_init__'s source and pins the ORDERING -- gate present, after _handle_uneven_tp, after the PD hook. Two can-fail arms executed: moving the call up next to the PD hook turns test_gate_runs_after_uneven_tp_resolves_dcp_size red, and dropping the page_size precondition turns the contract test and its subtest red. The first is the trap the placement correction exists for. Base-vs-branch on test/registered/unit/server_args/ + unit/disaggregation/ with the same command: 762 passed, 0 failed on branch, so no regression is possible. ruff clean on both changed files (server_args.py carries 355 pre-existing ruff findings, identical on base -- verified against HEAD:server_args.py); codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
Desk work for the gated GPU window: the wiring written down before it is built, and the criterion the wiring will be held to. No code changes. The lift criterion is a checkable table, L1-L9, each with the instrument that decides it. L6 is the one worth arguing for: acceptance rate on the PD pair must sit within the same-boot noise floor of the monolithic arm. A draft KV that is wrong in a way that does not crash shows up ONLY as lower acceptance, which reads as "this rig is slow" rather than as a defect -- the same silent-degradation class as #631a's original auto-disable. Lifting the refusal on L5 (same accepted-token sequence) alone would reintroduce it one layer down. W0 was read-only, so it was executed now rather than left as a step for the window, and it changes the recommendation. The draft pool's head count is decided by ONE predicate, read by both the pool sizing and the attention backend so they cannot disagree (owner.py:102 draft_pool_is_replicated). Under --draft-kv-layout replicated, the default and what production runs, the draft pool is HEAD-SHARDED, which is uneven under --rank-tp-ratio -- #631a's stated reason is correct there. Under --draft-kv-layout dcp the draft pool takes "the identical target-model treatment": full replicated kv-heads, token-sharded by the same weighted owner rule (model_runner_kv_cache_mixin.py:2783-2791). The second case dissolves the problem rather than solving it. A draft pool shaped exactly like the target pool crosses the PD boundary by exactly the mechanism that already carries the target pool -- the extra-layers registration on shared token indices (prefill.py:186, decode.py:439) plus the owned_ordinals scatter the R6 verdict established as already wired. No canonical shipment, no head reslicing, nothing new on the wire. And production sits inside sgl-project#108's covered shape item for item: NEXTN, topk 1, one draft layer, rank_tp_ratio installed, dcp_size 3 (server_args.py:7784-7799). So the recommendation is now path B first -- require --draft-kv-layout dcp for a PD pair that wants speculation -- reusing sgl-project#108's owner-rule kernels and the existing scatter instead of adding a third transfer shape. This partly supersedes the contract landed in 214433e, and saying so is worth more than defending it. That contract is not wrong and is not deleted: it remains the answer for the default replicated (head-sharded) layout, its geometry gate and version discipline are layout-agnostic, and local_head_window is already the recorded workaround for sgl-project#641. Retiring a proven desk artifact on an unproven expectation is how the fallback goes missing exactly when it is needed, so path B gets demonstrated on-card first. Path B is not free either, and the document says what it still owes: that the draft pool's DCP treatment is identical at the TRANSFER boundary and not merely similar (the extra-layers registration assumes matching per-rank item lengths, and sgl-project#345 is the standing example of a stride mismatch that corrupts silently), that --draft-kv-layout dcp composes with disaggregation_mode at all (no boot has combined them), and the measured acceptance cost.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 7, 2026
…t, yet Desk answer to the owed question, so the GPU window confirms instead of investigates. Documentation only. COMPOSITION VERDICT: UNCONSIDERED. draft_kv_layout appears in server_args.py exactly twice -- the argument definition (:1517) and the sgl-project#108 gate (:7799) -- and the sgl-project#108 gate contains no reference to disaggregation, prefill or decode. pd_disaggregation_hook.py contains no reference to draft_kv_layout. Only two files mention both concepts at all and in both the mention is incidental. Nothing refuses the combination and nothing supports it. WHAT BREAKS FIRST, from code rather than prediction. Three facts compose into a silent corruption. (1) The transport is draft-blind: mooncake/conn.py has ZERO occurrences of "draft", and the draft pool rides as ordinary extra layers under the comment "The indices are always shared with a target model" (prefill.py:186-195, decode.py:439-448). (2) The decode arm rewrites kv_indices into compact owner-rule rows ONCE, and that single array addresses every registered buffer, target and draft alike (decode.py:1119-1125). (3) The draft pool's row space depends on the layout: model_runner_kv_cache_mixin.py:2821-2823 returns the FULL global context for the default 'replicated' ("keeps the early return ... byte-identical") and falls through to dcp_compact_pool_rows only under 'dcp'. So on a DCP decode arm with the default draft layout, compact row indices are applied to a pool sized to the full global context -- two different coordinate systems, every draft row at the wrong address, the offset growing with the slot id. That is sgl-project#345's right-token/wrong-slot drift, silent rather than a crash. It is latent today only because #631a nulls speculative_algorithm on a PD arm, so no draft worker exists and the registration never runs. This is the strongest argument for path B so far, and it is not about efficiency: under --draft-kv-layout dcp the draft pool takes dcp_compact_pool_rows, the target's own coordinate system, so the shared indices are correct BY CONSTRUCTION. Path B is the only one of the two layouts on which the existing shared-index transfer is correct at all. Owed item (1) resolves with it: under 'dcp' the draft pool carries get_total_num_kv_heads() (:2788-2791), matching the target's treatment under uneven_dcp_kv_replicated, so per-rank item lengths agree by the same construction. NEW REFUSAL UNCOVERED, added to the criterion as L10 and L11: the two arms must agree on --draft-kv-layout, and a DCP decode arm must be on 'dcp'. Guard 1 cannot cover the first -- draft_kv_layout is deliberately not part of compute_model_identity_hash, being a parallelism decision rather than a weights one. Its natural home is beside the layout version in DraftKvCanonicalLayout, which is a second reason that contract stays.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 25, 2026
…he flip (3 front switches, all default off) MEASURED, boots weg2xsn429/430/432/433/435 (45 P-ladder requests, front log, tool /root/.claude/jobs/1ab4cd30/tmp/flip_roundtrip_27b.py). The ~4 s the ladder reads as "client wall minus P wall" (median 3.95-4.24 s) is a ROUND TRIP, not one flip: a route -> D->P flip begin 74-133 ms controller tick (0.2 s poll) b D->P flip (front) 1.56-1.60 s c D->P done -> P leg 1 start 199-203 ms controller tick, EVERY request -- P prefill (not counted) -- f P->D flip (front) 1.79-1.88 s g P->D done -> D first token 215-275 ms D readmission (eager extend) P end -> D first token (Flipzeit) 2.01-2.20 s Inside b, between the kv wake answering and `done`, the residue reading (`ps` + `nvidia-smi` as blocking subprocesses on the front's event loop) costs 44-48 ms (median per boot) after P is already awake; in f it costs 56-67 ms but is off the first-token path there. None of the three touches f/g: the dormant D admission already posts leg 2 during the P->D flip. The switches (env, read once at front start, named in one WEG2-FLIPFAST line): * SGLANG_WEG2_CTL_KICK_ARRIVAL=1 (F2): a request joining the P queue -- the four enqueue sites of handle_generate / leg2 reroute / W50 requeue -- and D's LAST leg 2 ending while the P queue holds work wake the controller at once. * SGLANG_WEG2_CTL_KICK_AFTER_FLIP=1 (F3): a completed D->P flip wakes it at once, so P's drain starts right after the flip instead of one tick later. * SGLANG_WEG2_DC_OFF_PATH=1 (F1): once nothing gates on the residue reading (W19 settled at the first D sleep with a non-empty reading, the group's dormant image sampled), it runs in the default executor after the flip closed; same WEG2-DC lines, the flip record's dc_mib is filled in place (dc_off_path=True marks it), plus one WEG2-DC-OFFPATH line. The first sleep of each group and every D sleep after an empty first reading stay synchronous. A kick only ends ONE wait early; every latch (economics, dwell, fairness, hand-off window) is evaluated exactly as on a tick. No kick may race d_admitter: the after-flip kick follows only a D->P flip (after P->D nothing waits on the controller), and an arrival kick is held while prefilled requests still wait in _ready_for_d (the D->P decision does not read it). Off = asyncio.sleep(0.2) and the inline reading, byte for byte. The literal `self.queue.append` stays (test_weg2_no_route_1290 pins it); Fronts built via __new__ in other suites read as all-off. Expected on the metal with all three on: client-P -0.33 s per P-routed request (a+c+dc: 0.086+0.203+0.046 s on xsn435), Flipzeit P-end -> first token unchanged (2.0-2.2 s; its levers are the legs and the readmission). Tests (desk, CUDA_VISIBLE_DEVICES="", via test27b.sh): * test/registered/unit/weg2/test_weg2_flipfast_27b.py, 20 tests: red on 9f0ac9c (15 failed / 5 passed -- the 5 are the default-path, W19 and source-pin guards that must pass there), green on this tree (20 passed). * every suite that imports the front (58 files): 1342 passed; 32 failed, and all 32 fail identically on 9f0ac9c (known red: 4 launcher argv tests on a SimpleNamespace without weg2_weight_source, the flip_cost '"tags": family' pin, the xsn288 weight_updater pin, no_route_1290 end-to-end on a __new__ Front without .vision, parked_abort_w1b "no current event loop", and 22 in sched_fix3/fix7/slice_a/rpc_conn/s3s4/text_only/w101/wake_kv_early/wcode/ x_seed). 0 new failures. NF line (H71, onto c701dd4 + H62): picked with -x, three conflicts in weg2/front.py resolved line by line, nothing else changed. - Module level: only the flipfast block (the three CTL_*/DC_OFF_PATH switches, CTL_TICK_S, CTL_KICK_REASONS, _env_switch_on) taken; its 27B neighbours _sleep_gate_by_name / _measured_record_stat_key (W100 sleep gate, not on NF and ordered off there) left out. - flip(): the F1 if/else wraps NF's own inline residue reading, kept verbatim and re-indented under `else:` -- dormant_residue_over and the --weights-resident W19 exemption (fnFL2 v21) stay in the synchronous branch, which is the first D sleep. NF's sample_dormant_image samples once per group as on 27B, so _dc_reading_deferrable holds unchanged. - Methods: the flipfast methods follow NF's _adopt_first_flip (sgl-project#108); the controller keeps `await self._adopt_first_flip()` before its loop, the loop's tick (the same `await asyncio.sleep(0.2)` as on 27B) is _ctl_wait(). H61's client_io / _STREAM_FINISH_RE are outside every hunk and unchanged. The P-intake-stall requeue (`appendleft`, weg2xsn272) gets no kick, as on 27B. (cherry picked from commit 502e97e)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 29, 2026
…ung -- Deckung ueber alle Tags + Riegel fuer ungedeckte Tensoren Prior-Art statt zweiter Mechanik: D ohne Platte gibt es seit sgl-project#108/sgl-project#109 (22.09., --weg2-d-adopt on: D laedt --load-format dummy, W115 sperrt jeden Forward, der Erstflip bringt Dense UND Experten-Praefix ueber die Legs -- die Platztausch-Puffer sgl-project#135 stehen in Manifest/Plan/Coverage --, kalte Zeilen kommen aus dem Store sgl-project#109). Am Metall zuletzt w87 (22.09.): 0 Presplit-Zeilen auf D. Seitdem sammelt der Wake je TAG (sgl-project#1374), und genau dort war die Naht offen: * _weg2_wake_reload_weights kehrte im per-Tag-Zweig ZURUECK, bevor der sgl-project#108-Block lief -- mark_adopted wurde nie gerufen, W115 fiel nie. * _weg2_last_inject_cover hielt nur den LETZTEN Tag (filled eines Tags gegen expected des ganzen Plans) -- selbst erreicht haette die Deckung nie gereicht. * Die Deckung zaehlte nur Plan-Deskriptoren: ein Tensor ohne Deskriptor (Peer haelt ihn nicht) behielte still seine Dummy-Bytes, waehrend der Riegel faellt. Fix: _weg2_adopt_accumulate sammelt je Tag (filled je Tag, expected = ganzer Plan, geschriebene Namen ohne ZEROFILL), _weg2_adopt_settle laeuft in BEIDEN Wake-Formen, _weg2_adopt_uncovered vergleicht jeden Tensor des Rangs (_weg2_rank_param_table) mit den geschriebenen Namen; SGLANG_WEG2_D_ADOPT_UNCOVERED=refuse (Default unter Adoption) laesst den Riegel dann stehen, =log nennt nur. Ohne --weg2-d-adopt ist alles inert (accumulate/settle tun nichts, solange keine Platzhalter stehen). Marker: 'sgl-project#108 ADOPT-COVER filled=N expected=N uncovered=0 -> PLATZHALTER GELOEST'; rot: 'RIEGEL BLEIBT', 'sgl-project#108 ADOPT-UNCOVERED n=.. first=..'. Test test_adopt_per_tag_bootzeit3_0929 (7): auf 4e153b8 rot (API fehlt), gruen mit Fix; Nachbarn adopt_erstboot/repack_adoption/store_als_quelle gruen. weg2-Wake-Suiten (flip_legs_h11, xchg_* u. a.) sammeln am Desk nicht ('Invalid device id', identisch auf der Basis).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 29, 2026
…ung -- Deckung ueber alle Tags + Riegel fuer ungedeckte Tensoren Prior-Art statt zweiter Mechanik: D ohne Platte gibt es seit sgl-project#108/sgl-project#109 (22.09., --weg2-d-adopt on: D laedt --load-format dummy, W115 sperrt jeden Forward, der Erstflip bringt Dense UND Experten-Praefix ueber die Legs -- die Platztausch-Puffer sgl-project#135 stehen in Manifest/Plan/Coverage --, kalte Zeilen kommen aus dem Store sgl-project#109). Am Metall zuletzt w87 (22.09.): 0 Presplit-Zeilen auf D. Seitdem sammelt der Wake je TAG (sgl-project#1374), und genau dort war die Naht offen: * _weg2_wake_reload_weights kehrte im per-Tag-Zweig ZURUECK, bevor der sgl-project#108-Block lief -- mark_adopted wurde nie gerufen, W115 fiel nie. * _weg2_last_inject_cover hielt nur den LETZTEN Tag (filled eines Tags gegen expected des ganzen Plans) -- selbst erreicht haette die Deckung nie gereicht. * Die Deckung zaehlte nur Plan-Deskriptoren: ein Tensor ohne Deskriptor (Peer haelt ihn nicht) behielte still seine Dummy-Bytes, waehrend der Riegel faellt. Fix: _weg2_adopt_accumulate sammelt je Tag (filled je Tag, expected = ganzer Plan, geschriebene Namen ohne ZEROFILL), _weg2_adopt_settle laeuft in BEIDEN Wake-Formen, _weg2_adopt_uncovered vergleicht jeden Tensor des Rangs (_weg2_rank_param_table) mit den geschriebenen Namen; SGLANG_WEG2_D_ADOPT_UNCOVERED=refuse (Default unter Adoption) laesst den Riegel dann stehen, =log nennt nur. Ohne --weg2-d-adopt ist alles inert (accumulate/settle tun nichts, solange keine Platzhalter stehen). Marker: 'sgl-project#108 ADOPT-COVER filled=N expected=N uncovered=0 -> PLATZHALTER GELOEST'; rot: 'RIEGEL BLEIBT', 'sgl-project#108 ADOPT-UNCOVERED n=.. first=..'. Test test_adopt_per_tag_bootzeit3_0929 (7): auf 4e153b8 rot (API fehlt), gruen mit Fix; Nachbarn adopt_erstboot/repack_adoption/store_als_quelle gruen. weg2-Wake-Suiten (flip_legs_h11, xchg_* u. a.) sammeln am Desk nicht ('Invalid device id', identisch auf der Basis).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #107
Co-authored with @johndun