Skip to content

Support stream=True in v1/completions - #49

Merged
merrymercy merged 6 commits into
mainfrom
cody/openai-completion
Jan 19, 2024
Merged

merrymercy merged 6 commits into
mainfrom
cody/openai-completion

Conversation

@comaniac

@comaniac comaniac commented Jan 18, 2024 •

Copy link
Copy Markdown
Contributor
  • Make OpenAI API v1/completions protocol compatible with OpenAI API.
  • Support stream=True in OpenAI API v1/completions.
  • Change the delimiter to \n\n so that it could be compatible with other frameworks.
  • Add unit tests.

On my local, all tests in tests/srt/test_httpserver_* and tests/srt/test_openai_server.py are passed.

@comaniac
comaniac force-pushed the cody/openai-completion branch from 6d9fbec to a6acdf3 Compare January 18, 2024 22:58
@merrymercy

Copy link
Copy Markdown
Collaborator

This looks good. Let me take this over and make the frontend compatible with your new changes

@comaniac

Copy link
Copy Markdown
Contributor Author

Sure. Feel free push necessary changes, and I'll rebase the chat completion API once this PR is merged.

@merrymercy
merrymercy merged commit 61d4c93 into main Jan 19, 2024
@merrymercy
merrymercy deleted the cody/openai-completion branch January 19, 2024 01:01
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
chunyuan-w pushed a commit to chunyuan-w/sglang that referenced this pull request Apr 15, 2025
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
* Update cython_setup.py

* Update __init__.py
pi314ever added a commit to pi314ever/sglang that referenced this pull request Aug 14, 2025
* Fix sglang llama4 rotary embedding and introduce mm inputs to hpu forward batch

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Enable working llama4 image processing, accuracy issue present

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Fix vision rotary embedding forward

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Bring in models to use native sglang classes

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Revert "Bring in models to use native sglang classes"

This reverts commit d61f0e7b68e52591b853ace7b79179a6b22940ea.

* Add disable cuda graph to disable hpu graphs

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Convert ndarray to tensors too

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Fix mm accessing

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Multimodal graph mode workaround

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* remove print statement

* ruff fix autofixable

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Fix syntax issue

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Fix rot emb no-vllm logic

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Fix is_cpu references in mllama4

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Minor fixes broken from rebase

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* isort and black formatting

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* Allow batched multimodal

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

* precommit format

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>

---------

Signed-off-by: Daniel Huang <daniel1.huang@intel.com>
Co-authored-by: Yang Wang <y3wang@habana.ai>
intervitens pushed a commit to intervitens/sglang that referenced this pull request Aug 31, 2025
key4ng pushed a commit to key4ng/sglang that referenced this pull request Nov 9, 2025
apinge referenced this pull request in apinge/sglang Dec 3, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
wisclmy0611 pushed a commit that referenced this pull request Apr 7, 2026
* docs: improve information density

* tighten information density

* fixed broken links

* docs: sync LMSYS SGLang blog cards

* fix sidebar

* feat: docs restructure

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 16, 2026
Two GGUF-specific bugs, one file.

1. (sgl-project#49) GGUF-MTP draft path over-reserved KV 5x: max_total_num_tokens
   100032 instead of ~470k. Root cause: _config_draft_model() sets
   num_nextn_predict_layers=1 on hf_config, but for Qwen3.5 that is the
   multimodal WRAPPER config while _derive_model_shapes() reads the
   field from hf_text_config (inner text config, which only carries
   mtp_num_hidden_layers) -> draft reports None -> model_runner falls
   back to the full target depth (num_hidden_layers=64) for draft KV
   sizing -> spec KV cell scaled by (1 + 64/16) = 5x. FP8 never hits
   this (embedded NEXTN head, no --speculative-draft-model-path).
   Fix: fall back to hf_config when the text config lacks the field
   (fires only when a draft rewrite set it; targets stay None).
   Measured: GGUF-MTP max_total 100032 -> 470912 (4.71x, per-rank
   [242698, 125088, 125209]; FP8 parity 475200).

2. (sgl-project#52) GGUF-only early-boot corruption: on a fresh boot, prompts
   >=64 tokens corrupted from ~token 2 until one /flush_cache healed
   permanently. M16's radix framing was wrong: victims had
   cached-token: 0, and order-swap showed victim selection is prompt
   length >=64 (mamba chunk grid), not request ordinal. Actual chain:
   sglang's server warmup sends a real image request whenever the model
   advertises image understanding -> the GGUF checkpoint has ZERO
   vision tensors (llama.cpp keeps vision in a separate mmproj file;
   the loader maps text+NEXTN only) -> warmup runs an UNINITIALIZED
   vision tower -> NaN image features contaminate that request's
   GDN/mamba states -> NaN residue in freed/recycled mamba slots ->
   every >=64-token prefill (track path) picks it up. FP8 runs the same
   path with real vision weights (finite residue, invisible).
   /flush_cache heals via mamba_pool.reset_state() (KV zeroing
   exonerated via SGLANG_FLUSH_ZERO_KV=0 control; --skip-server-warmup
   boot fully clean, isolating the source).
   Fix: force enable_multimodal=False for GGUF checkpoints (a
   checkpoint without vision weights is not an image-understanding
   model) -- warmup and API take the text path; image requests can no
   longer NaN-poison mid-serving either. FP8 multimodal untouched.

Validation:
- GGUF no-MTP: max_total 840896 unchanged; worst-case request order
  (74-tok victim first) all clean, NO flush needed.
- GGUF MTP fresh boot: max_total 470912; request sequence clean without
  flush; Paris + 17x3 pass; 1024-tok greedy long-form clean; conc-4 4/4
  clean; accept_len 2.48-3.44 (matches M17 content dependence).
- FP8-MTP regression: max_total exactly 475200, coherence pass, warmup
  still the image-chat path -> no cross-quant damage.

Open: real GGUF image support would need mmproj loading (rejected
cleanly for now); latent deferred-clear gap on recycled mamba track
slots is harmless with finite residue but would resurface with any
future NaN source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 16, 2026
GGUF checkpoints follow the llama.cpp convention of storing the vision
encoder in a separate mmproj*.gguf next to the text backbone. Until now
this fork skipped vision tensors entirely and (since 770fbb4) forced
GGUF to text-only. Port the vLLM-fork plugin's mmproj support so a
qwen35-family GGUF with an mmproj file keeps multimodal ENABLED and
loads a fully initialized vision tower; GGUF without mmproj keeps the
text-only fallback unchanged.

- gguf_qwen35.py (+168): detect_gguf_multimodal() (plugin glob rule:
  mmproj.gguf / mmproj-*.gguf / *mmproj*.gguf next to the backbone),
  clip->HF vision name maps ported from the plugin's qwen35 weights
  adapter, build_vision_name_map() + vision_weights_iterator() -- dense
  mmproj reader (gguf-py returns BF16 as raw uint8 -> view(bfloat16);
  F16/F32 shaped; llama.cpp's temporal Conv3d split v.patch_embd.weight
  / .weight.1 re-stacked at dim 2; unmapped mmproj tensor = hard error).
  Draft (NEXTN) configs never load vision.
- loader.py (+15): GGUFModelLoader chains the vision iterator after
  transform_stream() iff mmproj_path is present.
- model_config.py: inside the GGUF multimodal gate, a qwen35-arch
  ForConditionalGeneration with detected mmproj keeps multimodal
  enabled (logged); every other case takes the exact previous
  text-only disable path.

Sharding: NO replication -- mmproj names are normalized to exactly the
HF names the FP8 checkpoint loads (fused model.visual.*attn.qkv), so
GGUF vision inherits the fork's proven uneven-TP whole-head split
(6/5/5 over TP=3 auto). Offline proof: 333 emitted tensors == FP8's
333 visual tensors (names+shapes), values BIT-EXACT vs the FP8 vision
weights (maxdiff 0.0).

Validation (unsloth Qwen3.6-27B-MTP-GGUF + mmproj-BF16.gguf, tp=3
--rank-tp-ratio auto, weighted DCP, fp8 KV, ctx 32768):
- eager no-MTP: max_total 840896 unchanged; VLM warmup runs the real
  vision path; test image (red circle, "STOP") described correctly:
  shape circle, bright red, text STOP in white capitals; Paris/17x3/
  >=64-token prompts clean WITHOUT flush (old NaN victim class dead).
- MTP + CUDA graphs: max_total 470912 (no collapse of the sgl-project#49 fix);
  image correct; 1024-token long-form clean; conc-4 (1 image + 3 text)
  all clean; accept_len 2.08-3.40 with graphs on.
- FP8-MTP graphs regression: max_total exactly 475200, coherence
  clean, zero GGUF-multimodal log lines -> non-GGUF path untouched.
- GGUF without mmproj: identical text-only log + is_multimodal=False
  (offline ModelConfig check).

Open: huihui/heretic f16 mmproj variants verified offline (detection +
decode) but not live-booted; vision runs eager (same as FP8 today).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
Two GGUF-specific bugs, one file.

1. (sgl-project#49) GGUF-MTP draft path over-reserved KV 5x: max_total_num_tokens
   100032 instead of ~470k. Root cause: _config_draft_model() sets
   num_nextn_predict_layers=1 on hf_config, but for Qwen3.5 that is the
   multimodal WRAPPER config while _derive_model_shapes() reads the
   field from hf_text_config (inner text config, which only carries
   mtp_num_hidden_layers) -> draft reports None -> model_runner falls
   back to the full target depth (num_hidden_layers=64) for draft KV
   sizing -> spec KV cell scaled by (1 + 64/16) = 5x. FP8 never hits
   this (embedded NEXTN head, no --speculative-draft-model-path).
   Fix: fall back to hf_config when the text config lacks the field
   (fires only when a draft rewrite set it; targets stay None).
   Measured: GGUF-MTP max_total 100032 -> 470912 (4.71x, per-rank
   [242698, 125088, 125209]; FP8 parity 475200).

2. (sgl-project#52) GGUF-only early-boot corruption: on a fresh boot, prompts
   >=64 tokens corrupted from ~token 2 until one /flush_cache healed
   permanently. M16's radix framing was wrong: victims had
   cached-token: 0, and order-swap showed victim selection is prompt
   length >=64 (mamba chunk grid), not request ordinal. Actual chain:
   sglang's server warmup sends a real image request whenever the model
   advertises image understanding -> the GGUF checkpoint has ZERO
   vision tensors (llama.cpp keeps vision in a separate mmproj file;
   the loader maps text+NEXTN only) -> warmup runs an UNINITIALIZED
   vision tower -> NaN image features contaminate that request's
   GDN/mamba states -> NaN residue in freed/recycled mamba slots ->
   every >=64-token prefill (track path) picks it up. FP8 runs the same
   path with real vision weights (finite residue, invisible).
   /flush_cache heals via mamba_pool.reset_state() (KV zeroing
   exonerated via SGLANG_FLUSH_ZERO_KV=0 control; --skip-server-warmup
   boot fully clean, isolating the source).
   Fix: force enable_multimodal=False for GGUF checkpoints (a
   checkpoint without vision weights is not an image-understanding
   model) -- warmup and API take the text path; image requests can no
   longer NaN-poison mid-serving either. FP8 multimodal untouched.

Validation:
- GGUF no-MTP: max_total 840896 unchanged; worst-case request order
  (74-tok victim first) all clean, NO flush needed.
- GGUF MTP fresh boot: max_total 470912; request sequence clean without
  flush; Paris + 17x3 pass; 1024-tok greedy long-form clean; conc-4 4/4
  clean; accept_len 2.48-3.44 (matches M17 content dependence).
- FP8-MTP regression: max_total exactly 475200, coherence pass, warmup
  still the image-chat path -> no cross-quant damage.

Open: real GGUF image support would need mmproj loading (rejected
cleanly for now); latent deferred-clear gap on recycled mamba track
slots is harmless with finite residue but would resurface with any
future NaN source.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
GGUF checkpoints follow the llama.cpp convention of storing the vision
encoder in a separate mmproj*.gguf next to the text backbone. Until now
this fork skipped vision tensors entirely and (since 39176df) forced
GGUF to text-only. Port the vLLM-fork plugin's mmproj support so a
qwen35-family GGUF with an mmproj file keeps multimodal ENABLED and
loads a fully initialized vision tower; GGUF without mmproj keeps the
text-only fallback unchanged.

- gguf_qwen35.py (+168): detect_gguf_multimodal() (plugin glob rule:
  mmproj.gguf / mmproj-*.gguf / *mmproj*.gguf next to the backbone),
  clip->HF vision name maps ported from the plugin's qwen35 weights
  adapter, build_vision_name_map() + vision_weights_iterator() -- dense
  mmproj reader (gguf-py returns BF16 as raw uint8 -> view(bfloat16);
  F16/F32 shaped; llama.cpp's temporal Conv3d split v.patch_embd.weight
  / .weight.1 re-stacked at dim 2; unmapped mmproj tensor = hard error).
  Draft (NEXTN) configs never load vision.
- loader.py (+15): GGUFModelLoader chains the vision iterator after
  transform_stream() iff mmproj_path is present.
- model_config.py: inside the GGUF multimodal gate, a qwen35-arch
  ForConditionalGeneration with detected mmproj keeps multimodal
  enabled (logged); every other case takes the exact previous
  text-only disable path.

Sharding: NO replication -- mmproj names are normalized to exactly the
HF names the FP8 checkpoint loads (fused model.visual.*attn.qkv), so
GGUF vision inherits the fork's proven uneven-TP whole-head split
(6/5/5 over TP=3 auto). Offline proof: 333 emitted tensors == FP8's
333 visual tensors (names+shapes), values BIT-EXACT vs the FP8 vision
weights (maxdiff 0.0).

Validation (unsloth Qwen3.6-27B-MTP-GGUF + mmproj-BF16.gguf, tp=3
--rank-tp-ratio auto, weighted DCP, fp8 KV, ctx 32768):
- eager no-MTP: max_total 840896 unchanged; VLM warmup runs the real
  vision path; test image (red circle, "STOP") described correctly:
  shape circle, bright red, text STOP in white capitals; Paris/17x3/
  >=64-token prompts clean WITHOUT flush (old NaN victim class dead).
- MTP + CUDA graphs: max_total 470912 (no collapse of the sgl-project#49 fix);
  image correct; 1024-token long-form clean; conc-4 (1 image + 3 text)
  all clean; accept_len 2.08-3.40 with graphs on.
- FP8-MTP graphs regression: max_total exactly 475200, coherence
  clean, zero GGUF-multimodal log lines -> non-GGUF path untouched.
- GGUF without mmproj: identical text-only log + is_multimodal=False
  (offline ModelConfig check).

Open: huihui/heretic f16 mmproj variants verified offline (detection +
decode) but not live-booted; vision runs eager (same as FP8 today).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 19, 2026
…inite

Task sgl-project#49 (19.09.): three 262k boots answered '!!!' x768 (token 0 = NaN
logits) -- fn7n (no-enable-multimodal + INT8 mixers), fn7u (the flag
alone), fn7w (the INT8 mixers alone) -- while fn7l/fn7v/fn7t matched the
needle at 259k; short prompts are always right. Whether that is two
causes or one non-deterministic edge, the log so far cannot say: nothing
reports where the numbers die. With SGLANG_NAN_GUARD=1 every Qwen4Exp
layer checks its MoE output once per forward (skipped under capture) and
logs the FIRST non-finite hit per layer with forward mode, rows and
context depth; later hits are counted. Off by default, zero cost.

Tests: tests/moe_offload/test_nan_guard.py (1); suite tests/moe_offload
551 passed / 43 failed (baseline 43).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 20, 2026
… SGLANG_PDL_ON_SM12=1

fn8c12: the Triton fused-gate router, launched with PDL and carrying its gdc_wait, still turned finite gate logits into whole-token NaN topk_weights on the RTX 5090 (ROUTER-ORIGIN ENTERS-AT-TOPK-WEIGHTS bad_logit_rows=0 bad_weight_tokens=48 pairs_per_bad_token=10); the same kernel isolated from any predecessor is clean, and with the Triton router off (fn8c10b) the boot had no event in 4 needles. On sm_120 the griddepcontrol barrier does not order the dependent's read -- the class CUTLASS documents for compute_120 builds. is_arch_support_pdl() now answers False for major >= 12 (one INFO line per process), which covers all eight launch_pdl sites; the opt-in keeps the A/B spellable.

Gate: tests/moe_offload 43 failed (baseline set), no new failure.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 20, 2026
… SGLANG_PDL_ON_SM12=1

fn8c12: the Triton fused-gate router, launched with PDL and carrying its gdc_wait, still turned finite gate logits into whole-token NaN topk_weights on the RTX 5090 (ROUTER-ORIGIN ENTERS-AT-TOPK-WEIGHTS bad_logit_rows=0 bad_weight_tokens=48 pairs_per_bad_token=10); the same kernel isolated from any predecessor is clean, and with the Triton router off (fn8c10b) the boot had no event in 4 needles. On sm_120 the griddepcontrol barrier does not order the dependent's read -- the class CUTLASS documents for compute_120 builds. is_arch_support_pdl() now answers False for major >= 12 (one INFO line per process), which covers all eight launch_pdl sites; the opt-in keeps the A/B spellable.

Gate: tests/moe_offload 43 failed (baseline set), no new failure.
(cherry picked from commit 0d33570)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 25, 2026
… -- proof obligations pinned + window stress test

Frage (Operator 24.09. ~21:10Z, NF-Sitz bestaetigt dieselben Zeilen): das dichte
Template jit_kernel/csrc/gemm/marlin/marlin_template.h legt sh_red auf sh_b
(:638-639) und schreibt im Epilog sh_red (thread_block_reduce :1526) direkt nach
cp_async_wait<0>() (:1513) ohne Blockbarriere. Braucht es die Barriere, die NF
im MoE-Zwilling eingebaut hat (b8c451a, SGLANG_MARLIN_EPILOGUE_SYNC)?

Befund (Desk): nein, kein LEBENDES Byte ist erreichbar. Die letzte Blockbarriere
einer Slice ist wait_for_stage() bei k == b_sh_wr_iters-2; die Operanden beider
letzter matmul() sind VOR ihr geladen (Schleifenform fetch(k+1) -> Barriere ->
matmul(k)), matmul liest nur Register. Der einzige sh_b-Lesezugriff danach ist
der Prefetch fetch_to_registers(b_sh_wr_iters, (pipe+1)%stages) fuer eine
Iteration, die es am Slice-Ende nicht gibt: break -> Epilog -> start_pipes()
laedt frag[0] hinter einer Barriere neu (oder Kernelende). sh_g_idx/sh_zp/sh_s/
sh_a liegen hinter max(sh_red, sh_b); cp.async ist beim Epilog keiner mehr
unterwegs (pred slice_iters >= stages, wait<stages-2>); write_result endet und
barrier_release beginnt mit __syncthreads, die naechste Slice kann keinen
sh_red-Leser ueberholen. Uebrig bleibt ein FORMALES Data-Race auf einem toten
Load (racecheck wuerde es melden), ohne Wirkung auf Ergebnisse. Die MoE-Schleife
hat dieselbe Form; NFs Metall-A/B fn8c2 (Barriere AN, 20.09.) zeigte die
sgl-project#49-NaNs trotzdem, Wurzel war PDL auf sm_120 (0d33570; 27B: f24c7fd).
Das dichte Template und gptq_marlin.cuh sind auf NF (f398b48) und 27B-Linie
byte-gleich; dieser Commit aendert keinen Kernel (kein JIT-Neubau je Rang).

Wer faehrt das dichte Template: 27B-Linie nur der DFlash2-W8-Draft auf D
(CompressedTensorsWNA16 -> apply_gptq_marlin_linear -> JIT gptq_marlin), TP0
sm_120 + TP1/TP2 sm_86 (xsn436 D: "dense Marlin linears" 0.88/0.41/0.41 GiB,
Marlin-bf16-Hinweis TP1/TP2); P baut den Draft, rechnet ihn nie
(DFLASH-PRODUCE-ON-P off). Ziel gdncov ist W8A8 -> int8_scaled_mm, kein Marlin.

Tests:
- test_marlin_epilogue_alias_0924.py (hermetisch, liest nur den Template-Text):
  die sieben Beweisbedingungen + "sh_red nur im Epilog". 8 passed. Mutanten-
  Probe (6 Mutationen der Vorlage: Load hinter die Barriere, sh_b-Lesen in
  matmul, Tail-Barriere von write_result weg, sh_g_idx in den Alias,
  unpraedizierter Fetch, kein Reload in start_pipes): jede von genau ihrem Test
  gefangen, Original gruen.
- test_marlin_epilogue_alias_gpu_0924.py (Fenster, TEST27B_GPU=1, eine Karte):
  haemmert den UNVERAENDERTEN Kernel an den Draft-Shapes je D-Rang (58,25,25)
  + 2 small-N/large-K, M 1/8/16/256/2048, Arme prod/fp32/bf16red/(atomic ab
  sm_90), 20 Wiederholungen; Gewichte 128+-4 (ein Alias-Treffer waere ~1.6 rms
  gross), Referenz = Marlins eigene bf16-Dequant. Rot bei nicht-finit oder
  Grenze; bitweise Lauf-zu-Lauf-Abweichung innerhalb der Grenze wird nur
  berichtet (sgl-project#190-Klasse, 4de61b5). JIT nur lesend: der Vorcheck faehrt den
  ECHTEN load_jit-Pfad mit ersetzten Schreibern (heal_entry/building_marker/
  purge_entry/_selfheal_jit_cache_once) und Sonden-load_module; Desk-Probe mit
  erzwungener Arch: 12.0 und 8.6 -> LOADS ~/.cache/tvm-ffi/...gptq_marlin_bf16_t_
  d6053cd6e445c563 (+ repack 7b12d3073e80e78d), gemeinsamer Cache unveraendert
  (mtime, 338 Eintraege); ohne GPU skip. GPU-Lauf: offen, Operator-Fenster.

(cherry picked from commit 38c008c)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…cks, chronologisch)

Grundlage: Präsenz-Scan aller 230 27B-Commits seit 76f8deb gegen diesen Baum
(Stichprobe der hinzugefügten Zeilen je Commit); die 94 fehlenden minus die bewusst
anders gewählten Formen (76e87ac/4ae11ababd -> S3 form.calibration_identity;
479f6ec/d7f588e017/d0fba8955f/34892e3017 -> S2 NF-Formen; 7f81f09/3c14481318 ->
S4/S7a; 3dbb790 line_gate_27b (Werkzeug, Schritt 9); 8604d13 W100-by-name
(Nutzer: bleibt aus); 6545e2c flashinfer-Pin in pyproject (Image-Frage, nicht Baum)).
Liste: 92bbccb eb5d044 829ebd0 431fcbc ef4d11f 6816062 a233e50 2cc593c f0c8451 87cc4fb c529777 31f2dbe c50085a 3301a96 036b368 e1d1fe9 03c68af 6dddc2e 06932b5 87389c4 58a7490 f85ac55 fc64aa5 5aa24dd 97c0e9a 159333c d9f1532 f3c685b 8550655 e50fb59 db2c2ef f09dc0d c255e10 51b810e 28a55a2 34965fc ff3d9cc 340a018 bee5e10 67b6352 fdade85 ed6630d f1c9a43 b434831 517f26d 0b6b60b a40837f 644de86 aff2b7c 197b701 856024b 238512a 9738626 b857a22 1f8c24d d294b3e 810239d b429dfd e714c95 9efd974 3d63e0a d3cfcf3 fee6134 7985b56 49a14e9 fc45706 19c720e 5306bee 6f1235a c98eaa3 93bc802 328349e f9fb3a2 572af73 94fa8b4 d342caa 2fd7d7e 3ebbb96 871d55f 78c2f16 3babf51 196f6a8 e70af54 22eccfc

Inhalt: Upstream-Ports (sgl-project#33758 sgl-project#37818 sgl-project#36738 sgl-project#33459/sgl-project#30096 sgl-project#34446 sgl-project#36267 sgl-project#33778 sgl-project#34859
sgl-project#36415 sgl-project#35255/sgl-project#36638 sgl-project#39858/sgl-project#40259 sgl-project#31417 sgl-project#34892 sgl-project#32225 sgl-project#30832/sgl-project#36626 sgl-project#39574 sgl-project#29579
sgl-project#31468 sgl-project#32575 sgl-project#31648); xsn409/410-Wake-Verdikte; Vision-Linie V1-V3b + xsn438
(SGLANG_WEG2_VISION_FLIP_URGENT); D-Planer L6 (159333c); DFLASH-Window-Pool
sync-frei, PLAN_SYNC_FREE, D-Kollektive (vocab-argmax, a2a-Merge, deferred rebuild),
#DGAP/D_DEFER_SEQ_LENS_CPU; Mamba-Anker Raster 4096 + Per-Path-Cap + Inner-Release;
P-TRIM (--p-trim-end-anchor); FP8 uniform Marlin; ModelOpt/NVFP4 RadixArk; GGUF G1-G6
+ F1/F2; native-mixed sgl-project#38 (sm_8x W4A8, sm_12x CUTLASS/W4A16); RC1-Capture-Set; sgl-project#49
Agent-Turns; dynchunk (--p-chunk-policy, --p-chunk-dynamic-min-tokens).

Auflösungen (Gabel -> Form, Grund):
- L6 d_operating_point_rows: 27B (d) "Token-Vektor auf jeder Position aus der Kapazität"
  nur bei TP-symmetrischem D (Profil d_layout paged_dcp); sonst NF-sgl-project#1293-Pin + NF-Anker-
  Klausel. mamba_ssm_dtype aus EARLY_READ_FACTS nur bei Profil early_read_flags.
  Overhead-Kalibrierung liest mit form.CalibrationIdentity statt LineIdentity.
- RC1 Capture-Set: neuer RecordKey-Term d_capture_set (qwen27b), Leser
  CalibrationIdentity.d_max_running_requests; nextflash unverändert.
- URC Carrier-Hold: 27B _weg2_carrier_hold entfällt (S2 NF-Rotation), Inner-Release und
  Per-Path-Cap bleiben (Env, Default aus; 27b.env setzt sie).
- scheduler_pp_mixin/overlap_utils/batch_result_processor: NF H49/H58 und 27B #PGAP/#DGAP
  komponiert (beide Instrumente getrennt schaltbar).
- schedule_policy: P-TRIM-Kurzschluss vor NF H63-Fold/QSA-Korn; sgl-project#36415 Hoist + NF
  computed_input_len.
- gdn_backend sgl-project#33778: 27B-strided-Verify; NF-Ring flacht beide Layouts ab.
- flashinfer_backend: RC9-Datei + NF-Form-A-Waiver (27B-intern mehrfach gegabelt).
- checkpoint_census: GGUF-Leser + NF exclude_segments (PLE) in einer Aggregation.
- xchg_manifest: FLAT_SEGMENTS in beiden (dst/src) NF-Breitenbedingungen ausgenommen.
- vram_peak_window: NF-Kumulativ-Peak liest über den 27B-Fast-Read.
- FP8: 8c86eb8-Rest nachgezogen (private Workspace-Registry entfernt, wie RC9).
- argv_d: vision= an allen drei Aufrufstellen (inkl. NF --d-only).
- census_checkpoint_decision (W161) jetzt für beide Profile aktiv.

Gates: py_compile aller geänderten Dateien; ruff F821/F811 ohne neue Funde gegenüber
dem Vorgänger (PendingSeqLensCpu ist String-Annotation wie in RC9); dup_defs_gate 0 neu.
Tests: 73 portierte Testdateien, Lauf nach dem Ruhefenster (Boot aktiv).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…: in-flight span credit only, sgl-project#49 itself was already in (S7c)

Operator 26.09.: FS ported sgl-project#49 (27B 196f6a8) onto c1a1a00 behind
SGLANG_WEG2_FRONT_SPAN_49, because sgl-project#49 was missing there. The unified tree
has had sgl-project#49 unswitched since S7c (9266605, pick list includes 196f6a8;
SpanLRU entries (text, ct, pt, held), record_presence(prompt_tokens,
held_epoch), tools first, test_weg2_front_span_agent_49.py present). So
only FS's in-flight half is taken here, with no second copy of sgl-project#49 and no
_49 switch:

- environ: SGLANG_WEG2_FRONT_SPAN_INFLIGHT = EnvBool(False)
- front: front_span_inflight(), SpanLRU.record_inflight, stream_has_content,
  and the leg-2 hook: first content event -> credit the text for this epoch
  (switch on); WEG2 LEG2-FIRST-CONTENT rid= epoch= via= leg2_ms= instrument
  (always on, routing-neutral). The code is verbatim from FS.
- test_weg2_front_span_inflight_fs.py: FS's file without the four
  _49-switch/off-path tests (tools-last, pre-port arithmetic, call-site
  source pins, inflight-without-_49), plus test_switch_default_off, which
  pins that no front_span_49 exists.

Gates: dup_defs_gate vs 27b-rc9-103712cdb2 + nf-rc2.1l-d1c7094ba6: 0 new.
test27b.sh: front_span_inflight_fs + front_span_agent_49 + store_consistency_1324: 43 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…NG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE, default off)

Claude Code 2.1.280 sends mid-conversation role:"system" messages
(api_system: tool additions/removals, output_config). With the Qwen3.8
template (head-only system turn) the Anthropic adapter hoisted them into the
head system turn, so the first turn that carries one re-renders every token
behind the system text. Measured on D: radix sgl-project#1420 WALK-STOP and the arena
key chain (sgl-project#1439 leading_complete) both stop at the end of the system turn
-- depth 3876 (dkr27bnvfp4bar1agent09252328, weg2-14-16/-17) and 4655
(dkr27bbar1final09260145, weg2-12-12), stored (13,248046)=".<|im_end|>",
new (13,198)=".\n" (the "\n".join of the hoist). The sgl-project#904 census then reads
MambaComponent:absent at that depth, X-GATE W31, X-REQUEUE, one P epoch.
The front's request_text renders the same message in place and credits the
whole previous turn (presence_span 19335/20168), so it routes SHORT.

Switch on: non-leading inline system messages become a user turn at their
position, wrapped in <system-reminder>; the leading run is still merged into
the head. Must be set identically on P and D (handoff keys are P's
tokenization). Off = byte-identical to before.

Tests (test27b.sh, CPU):
  test_inline_system_in_place.py 5 passed (off pins the break at ".\n",
  on pins turn N as a rendered prefix of turn N+1);
  anthropic/test_serving.py + test_anthropic_557.py + test_anthropic_rid_h90.py
  + weg2/test_weg2_messages_wire_rc7_0925.py: 100 passed together.

UNIFY (UN6, Operator 26.09.): pick of desk/27b-mambaend-0926 (2303948, MZ, base 0644c6f) after
the sgl-project#49-rest port, clean. dup_defs_gate: 0 new. test27b.sh: test_inline_system_in_place +
anthropic/test_serving + test_anthropic_557 + test_anthropic_rid_h90 91 passed, 9 subtests.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…le field -- qwen27b on, nextflash off (operator decision 26.09.; NF P49 c1988ff)

sgl-project#49 (27B 196f6a8) ran unswitched in the unified tree since S7c
(9266605), i.e. for nextflash too -- invisible in the byte-identical NF
argv. NF gates it behind SGLANG_WEG2_ENABLE_AGENT_SPAN (P49 c1988ff,
default off = rc2.1l + H100). Taken here with that commit's front.py and
tests (conflicts line by line: environ next to the FS/S7 entries, front's
request_text signature next to front_span_inflight), and the default made a
MODEL PROFILE field (ModelProfile.agent_span -> PROFILE_SWITCH_DEFAULTS):
qwen27b on (the 27B behaviour since RC9, unchanged), nextflash off until the
NF seat releases it with a boot tag, off without a form (the NF code
default). An explicit env value wins.

Off = P49's form: request_text renders tools last, record_presence drops
prompt_tokens/held_epoch, span_tokens is rc2.1l's, ROUTE-VERDICT without the
sgl-project#49 witness, no "WEG2 AGENT-SPAN sgl-project#49 on" line. Added: the sgl-project#49 rest
(SGLANG_WEG2_FRONT_SPAN_INFLIGHT, FS 021272a) credits only when the span
is on too -- record_inflight writes a held entry and would bring sgl-project#49's
held-epoch price back under span off (mutant: removing the gate turns
test_inflight_credits_only_under_the_span red).

Tests (test27b.sh, CPU): new test_unify_agent_span_profile.py 14 (rows,
per-profile default on env/SpanLRU/request_text, explicit wins, one environ
entry, nextflash drops the held epoch, FS gate both ways); P49's
test_weg2_agent_span_switch_p49 (default pin now via no form); the FS tests
set the span on (27B form). 25 front/span/profile files: 431 passed, 13
failed -- the 13 identical on 61d9241 (no_route_1290 x2, s3s4_1233,
sched_fix3 x4, scheduling_slice_a x6). profile_docker --check: AGENT_SPAN
registry-only in every profile (no .env sets it); the one DIFF is the new
measurement arm 27b-int8-x16 (--x-ceiling-tokens 16384, deliberate).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
Conflict in weg2/front.py (request_text gained tools_first next to the new
front_span_inflight_line): both kept. The sgl-project#49 armed line now also names
SGLANG_WEG2_ENABLE_AGENT_SPAN, which the in-flight credit needs as well
("WEG2-FRONT sgl-project#49 SPAN-INFLIGHT armed=<0|1> agent_span=<0|1> role=front pid=").

Tests (desk, test27b.sh): test_weg2_prefix_trace_in + store_told
1400/1416e/hold_tk/positive_tk, twin_defer, inline_system_in_place,
front_span_inflight_fs, arena_reset_release_h81, node_value_trail_1469,
producer_phase_census_wiring_1061, hicache_arena_claim_1427, handoff_1442,
arena_queue_refs_27b, p_intake_h91a, weg2/test_weg2_front*.py: 213 passed,
2 failed (test_weg2_front_lan_1321 TestFrontHostArgv x2 -- fail identically
on b3180fb itself, launcher.py weg2_weight_source).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153)

Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also
red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only.

- pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted
  the admission-decision arc the whole file drove (_pp_send/_recv_admission_
  decision, _pp_drain_voided_proxy, ...) and made the proxy channel
  unconditional; #1015c (206661a) retired the downstream refusal
  (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream
  pass refused, 0 served). The file now pins both halves of that retirement
  (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name).
- pp_upstream_void_before_formation_951 x3 (found in the same family): the
  three refusal cases rewritten to the #1015c form (pass reaches the
  formation, predicate False on every shape, no void -> streak 0, slot kept).
- scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency +
  SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p
  keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads
  ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the
  case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the
  P->D flip D legitimately admits into; it now models the D->P flip as flip()
  runs it (awake stays 'D' until the end). Mutant (post-acquire re-check
  removed) -> red. t3 is the front defect, next commit.
- sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst
  verdict into a local tested in the loop (b1d: locals assigned from a
  call/attribute count as state; b1e-g: the slice gets its stock values
  _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time`
  (real clock). Round-5 mutant (`if req is not None: continue` above the gate)
  still red on all four.
- no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49
  epoch keyword of price_remainder (196f6a8), H84 x_start_tokens
  (b895928).
- s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request
  around _handle_generate_request_impl; the seam is the impl's first
  statement, the wrapper is pinned to clock-read + delegation.
- pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants
  the usable-match floor on tree_cache; harness tree_cache=None.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…gl-project#49 armed lines

Every prefix miss of an agent-load boot gets a token receipt. With
SGLANG_WEG2_PREFIX_TRACE=1 (min rest SGLANG_WEG2_PREFIX_TRACE_MIN_TOKENS,
default 1024), read once per process (weg2/prefix_trace.py, armed line
"#PT PREFIX-TRACE armed=<0|1> min_tokens=M group=G pid=P"):

- sgl-project#1420 WALK-STOP: one line per (rid, stop depth) with rest >= min, no
  process cap, root stops included; appends
  "rid=<full> trace=1 stop_node=N stop_hash=H want_run=[..] have_run=[[..]]".
  A split stop now names the TRUE rest and want (the un-advanced key made
  depth+remaining exceed pt-1 by the split's matched part; also untraced).
- sgl-project#1400/sgl-project#1416/#1416d/#1416e (and TK's sgl-project#1442 ADOPT / TK ABS-TOLD): full rid,
  8/256 sampling lifted.
- sgl-project#1442 HANDOFF-KEYS REG, sgl-project#1040 EXTENT STATE-ALIGN: one line per distinct
  (rid, matched, span) / (rid, extent).
- sgl-project#1469 EVICT: uncapped (outside the 600), + parent, pages, hash_first,
  hash_last, key_len, trace=1.
- sgl-project#1427 ARENA-DROP: uncapped, + "trace=1 claim=<stem> dropped=<lo64 hex,..>".
- sgl-project#1439 ARENA-PRESENT: once per distinct (keys, leading, first stem) with
  rest >= min, + "trace=1 break_stem_full=<stem>" (joinable to dropped=).
No traced line sits on the decode round path.

Armed lines (once per process): "#MZ INLINE-SYSTEM-IN-PLACE armed= merge=
group= pid=" (anthropic adapter, P and D HTTP process) and "WEG2-FRONT sgl-project#49
SPAN-INFLIGHT armed= role=front pid=" (front start).

Tests (desk, CUDA_VISIBLE_DEVICES="", test27b.sh):
Base: desk/27b-unified-0926 @ 2bddf04 (rebased from f4b6f21, clean).
- new test/registered/unit/mem_cache/test_weg2_prefix_trace_in.py: 16 passed
- rebased run, all files below plus test_weg2_store_told_{hold,positive}_tk:
  324 passed
- managers/test_weg2_store_told_{1400,paced_1416e}, test_weg2_p_twin_defer,
  test_prefetch_deferral_1068, test_weg2_p_burst_clock_h42c,
  test_weg2_p_multi_anchor_tails_h42, anthropic/test_inline_system_in_place,
  mem_cache/test_arena_{queue_refs_27b,inner_anchor_release_0924},
  test_backup_ack_drain_tp_asymmetry_h74, test_hicache_drain_budget_27b,
  weg2/test_weg2_{p_intake_h91a,front_span_inflight_fs,arena_reset_release_h81,
  node_value_trail_1469}, test_producer_phase_census_wiring_1061: 232 passed
- test_hicache_arena_claim_1427, test_weg2_handoff_1442,
  test_weg2_store_short_tail_xsn437, test_evict_frees_the_bound_allocator_941,
  test_weg2_p_trim_end_anchor_0924: 50 passed
- test_unified_radix_cache_unittest.py, per class, against the base:
  identical FAILED sets (pre-existing on the CPU desk; ps128_sw4/sw128
  OOM in the 3 GiB cgroup on both trees)

Unify note (RM, pick onto 2a7f992): conflicts line by line (store_told: PF fallback forget kept + IN _rt(rid); front: IN line helper + the P49 request_text signature). The sgl-project#49 SPAN-INFLIGHT armed line also names agent_span (the inflight credit needs the sgl-project#49 span, profile field agent_span); its test pin follows.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…e), no 1.3*X band -- behind SGLANG_WEG2_FRONT_EXACT_TOKENS (default off)

User decision 26.09. ~19:00Z (memory d2p-sofort-flippen-und-x-exakt-0926): X holds
EXACTLY for the pending tokens; the 1.3*X SHORT band (release table row 15, an open
question, never built) is dropped.

Measured, boot dkr27brc10bar1agent09261821 (front.log, 114 requests): chars/3 over-
counted whole prompts by median 11.5 % (3.35 real chars/token, up to +60 %); D-direct
error median +11, |median| 125, range -4481..+4273; misroutes: 1 SHORT really > X
(weg2-10-7 3939/4780), 2 LONG really <= X (30-97, 30-106), ~6 LONG only by the chars/3
over-count. 9/78 D-direct under-priced by 1.0-4.7k: D resumes only at the Mamba anchor
(sgl-project#988 LOADBACK anchor_depth = P's cached of the predecessor) -- a D cache fact, logged
per request (X-EXACT-ERR), not banded.

On: weg2/front_tokens.py builds D's OWN rendering stack in the front (AnthropicServing +
OpenAIServingChat + TemplateManager) from the group's /get_server_info server args
(tokenizer_path, chat_template, chat_template_default_kwargs, parsers), so the count is
D's count_tokens code; pending = tokens - min(D's measurement, token LCP) (TokenSpans,
sgl-project#1324/sgl-project#49 semantics in tokens, epoch-bound held credit). One worker thread, incremental
per <|im_start|> segment (special token -> exact split). Lines: X-EXACT READY / PRICE /
FALLBACK (not ready, timeout, multimodal, path) / TOKENS (front vs group count) / ERR.
Off: nothing constructed or imported; ROUTE-VERDICT and X-ROUTE bytes unchanged.

Cost (desk, 27B INT8): 160k tokens cold 302 ms, next turn 55 ms, event-loop max lag
31 ms; load 7 s, RSS +608 MiB first load. NF INT4 160k: cold 251, warm 57 ms.

Tests: test_weg2_front_x_exact.py 21/21 (real tokenizers 27B INT8 + NF INT4-mixed:
count == D count_tokens on real-text Claude-Code payloads, segmented == whole on 300
hostile strings, MZ inline-system parity, boundary X/X+1, credit, fallback, residual
error log, switch off byte-identical). Affected suites (p49, sgl-project#49, inflight, messages
pricing/usage, idle policy, dp wait, flipfast, vision economics, H75/H78, no_route_1290,
x-split): 283 passed, 2 failed = test_weg2_no_route_1290 TheRouterEndToEnd x2, identical
red on base c51591f (Front has no attribute 'vision'). dup_defs_gate exit 0.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…e), no 1.3*X band -- behind SGLANG_WEG2_FRONT_EXACT_TOKENS (default off)

User decision 26.09. ~19:00Z (memory d2p-sofort-flippen-und-x-exakt-0926): X holds
EXACTLY for the pending tokens; the 1.3*X SHORT band (release table row 15, an open
question, never built) is dropped.

Measured, boot dkr27brc10bar1agent09261821 (front.log, 114 requests): chars/3 over-
counted whole prompts by median 11.5 % (3.35 real chars/token, up to +60 %); D-direct
error median +11, |median| 125, range -4481..+4273; misroutes: 1 SHORT really > X
(weg2-10-7 3939/4780), 2 LONG really <= X (30-97, 30-106), ~6 LONG only by the chars/3
over-count. 9/78 D-direct under-priced by 1.0-4.7k: D resumes only at the Mamba anchor
(sgl-project#988 LOADBACK anchor_depth = P's cached of the predecessor) -- a D cache fact, logged
per request (X-EXACT-ERR), not banded.

On: weg2/front_tokens.py builds D's OWN rendering stack in the front (AnthropicServing +
OpenAIServingChat + TemplateManager) from the group's /get_server_info server args
(tokenizer_path, chat_template, chat_template_default_kwargs, parsers), so the count is
D's count_tokens code; pending = tokens - min(D's measurement, token LCP) (TokenSpans,
sgl-project#1324/sgl-project#49 semantics in tokens, epoch-bound held credit). One worker thread, incremental
per <|im_start|> segment (special token -> exact split). Lines: X-EXACT READY / PRICE /
FALLBACK (not ready, timeout, multimodal, path) / TOKENS (front vs group count) / ERR.
Off: nothing constructed or imported; ROUTE-VERDICT and X-ROUTE bytes unchanged.

Cost (desk, 27B INT8): 160k tokens cold 302 ms, next turn 55 ms, event-loop max lag
31 ms; load 7 s, RSS +608 MiB first load. NF INT4 160k: cold 251, warm 57 ms.

Tests: test_weg2_front_x_exact.py 21/21 (real tokenizers 27B INT8 + NF INT4-mixed:
count == D count_tokens on real-text Claude-Code payloads, segmented == whole on 300
hostile strings, MZ inline-system parity, boundary X/X+1, credit, fallback, residual
error log, switch off byte-identical). Affected suites (p49, sgl-project#49, inflight, messages
pricing/usage, idle policy, dp wait, flipfast, vision economics, H75/H78, no_route_1290,
x-split): 283 passed, 2 failed = test_weg2_no_route_1290 TheRouterEndToEnd x2, identical
red on base c51591f (Front has no attribute 'vision'). dup_defs_gate exit 0.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…xtflash aus) -- PF als Feld, aus

Metall-Beleg dkr27brc10bar1agent09261821 (Image rc11a, 26.09. 18:21-18:56Z): 108 Requests,
Flips 0,23 je Request (vorher 1,35), verschwendeter Prefill ~7 % (vorher 48 %), 0 Gruppentod,
ENV-IM-RANG alle fuenf = 1 auf P, D und Front; #1416d TOLD-PROBE=8, PACED 8/8, TWIN 7/7,
span_inflight 109.

weg2/form.py: PREFIX_SWITCHES (Feld -> Env) und sechs ModelProfile-Felder
  inline_system_in_place  SGLANG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE (MZ, P==D Pflicht)
  told_probe_tree_key     SGLANG_WEG2_TOLD_PROBE_TREE_KEY (TK; TOLD_ABSOLUTE folgt als Rang-Default)
  told_paced              SGLANG_WEG2_TOLD_PACED (PX2)
  p_twin_defer            SGLANG_WEG2_P_TWIN_DEFER (TW)
  front_span_inflight     SGLANG_WEG2_FRONT_SPAN_INFLIGHT (sgl-project#49, wirkt nur mit agent_span)
  told_group_fallback     SGLANG_WEG2_TOLD_GROUP_FALLBACK (PF; auf beiden Zeilen AUS, am Metall ungeprueft)
qwen27b: die fuenf an, PF aus. nextflash: alle aus, bis NF freigibt. switch_defaults()
fuehrt sie, damit profile_docker --check Drift sieht.

Muster agent_span/standard_form: prefix_switch_state (explizite Env gewinnt, jede Schreibweise
der name_compat-Familie; sonst Registry-Zeile; ohne Form aus), publish_prefix_switches (der eine
Schreiber: build_env fuer P und D vor --env-p/--env-d, dieselbe Aufruf auf die Front-Env; ein
Aus-Wert schreibt NICHTS -> NF byte-gleich), launcher-Zeile WEG2-PREFIX-SWITCHES mit Wert und
Quelle (env | profile), Verweigerung bei MZ-Split ueber --env-p/--env-d. Rang-Seite: environ.py-
Eintraege mit _profile_default (Invariante S2/S3), die direkten Leser (weg2_store_told,
p_twin_defer, weg2_told_fallback) nehmen ihren Default ueber form.prefix_switch_armed, das
explizite Parsing bleibt.

Tests (test27b.sh, CUDA_VISIBLE_DEVICES=""):
  test_unify_prefix_switches_profile.py + agent_span/standard_form/S2/S3/S7/S8/S9 profile,
  build_env_group, name_compat_env_1b, form_axes, front_span_inflight_fs, told paced/pf/tk
  positive/hold, inline_system_in_place, prefix_trace_in: 313 passed.
  p_twin_defer, store_told_1400, prefetch_deferral_1068, p_burst_clock_h42c,
  p_multi_anchor_tails_h42, p_intake_h91a, launcher_argv_1235, vision_d_launcher,
  fp8_uniform_marlin, p_prefill_transient_114, rankdump_1378, hostsample_csv_1378: 200 passed.
Parse-Ebene (FL4-Form, main-Vorlauf + build_env P/D + Front-Env, Basis 84183ca):
  nf.env Basis gegen neu: byte-gleich bis auf die Boot-Nonce (194 Zeilen).
  27b-release-draft ohne _form-Zeilen (neu) == alte Fassung mit _form-Zeilen auf der Basis (172 Zeilen).
  27b-Gruppen-Env ohne _form-Zeilen: +5 Schalter je P, D, Front gegen die Basis.
profile_docker --check /spinning/gpu-arb/docker/profiles: TOTAL DIFF 1 (27b-int8-x16
  --x-ceiling-tokens 16384, vorbestehend, auf der Basis ebenso); release-draft/27b/nf 0 DIFF.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153)

Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also
red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only.

- pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted
  the admission-decision arc the whole file drove (_pp_send/_recv_admission_
  decision, _pp_drain_voided_proxy, ...) and made the proxy channel
  unconditional; #1015c (206661a) retired the downstream refusal
  (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream
  pass refused, 0 served). The file now pins both halves of that retirement
  (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name).
- pp_upstream_void_before_formation_951 x3 (found in the same family): the
  three refusal cases rewritten to the #1015c form (pass reaches the
  formation, predicate False on every shape, no void -> streak 0, slot kept).
- scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency +
  SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p
  keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads
  ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the
  case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the
  P->D flip D legitimately admits into; it now models the D->P flip as flip()
  runs it (awake stays 'D' until the end). Mutant (post-acquire re-check
  removed) -> red. t3 is the front defect, next commit.
- sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst
  verdict into a local tested in the loop (b1d: locals assigned from a
  call/attribute count as state; b1e-g: the slice gets its stock values
  _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time`
  (real clock). Round-5 mutant (`if req is not None: continue` above the gate)
  still red on all four.
- no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49
  epoch keyword of price_remainder (196f6a8), H84 x_start_tokens
  (b895928).
- s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request
  around _handle_generate_request_impl; the seam is the impl's first
  statement, the wrapper is pinned to clock-read + delegation.
- pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants
  the usable-match floor on tree_cache; harness tree_cache=None.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
… until measured), exact W31 re-queue, queue re-priced exactly for PK's needs_p

Follow-up to c0347b7 (X holds exactly for the pending tokens, user 26.09.; no 1.3*X band).

1. Registry: ModelProfile.front_exact_tokens -> SGLANG_WEG2_FRONT_EXACT_TOKENS via
   PROFILE_SWITCH_DEFAULTS / _profile_default (both rows False until an agent-load boot
   measured X-EXACT-ERR / X-EXACT-TOKENS; explicit env wins; off without a form).
2. The W31 re-queue site (third Pending construction) priced the whole prompt by
   len(text)//3; under the switch it takes the front tokenizer's exact count.
3. The arrival price goes stale (a sgl-project#49 held credit lives only in its epoch, an in-flight
   twin credit or a D serve lands after the arrival, a measured zero retracts):
   Front._x_exact_reprice_queue re-prices every queued request whose price can still
   move -- at the epoch change, after record_inflight and after record_presence --
   against the current TokenSpans. PK's phase_policy.needs_p / immediate_park_trigger
   read Pending.est_uncached, so the park fires on the CURRENT exact pending. Untouched:
   leg1_done, skip_leg1, p_only, x_requeues (D's own extent), chars/3-fallback prices.
   Only the number moves, routing flags do not. Line: WEG2 X-EXACT-REPRICE ... crossed=.
Not built (operator): the Mamba-anchor residual (D resumes at an anchor below the
credited prefix after an after_p serve) stays a logged X-EXACT-ERR, no front heuristic.

Tests: test_weg2_front_x_exact.py 35/35 (+14: registry rows off, profile row/explicit
matrix, twin credit reprices X+500 -> 200 and needs_p stops firing, epoch change raises
the price over X and needs_p fires, measured/final prices untouched, off never reprices,
W31 site exact). Affected suites (unify profile/switch/standard-form/anchor/prefix,
27B park immediate, p49, sgl-project#49, inflight, messages pricing/usage, idle policy, dp wait,
flipfast, vision economics, H75/H78, no_route_1290, x-split): 459 passed, 0 failed.
dup_defs_gate exit 0 (no new finding vs a4158ae).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 27, 2026
…stay off) -- P's store read of an NF session is no longer clamped to told=0 by a unigram anchor probe

NF rc12t (dkrnfh91dprbar1dauer09271756, e1eb275, profile nf-h91-dpr): 0 of 41 P legs cached. P's PP0 read
the store (weg2-1-9 18:05:34 "HiCache prefetch success ... loaded=54208", WEG2-LOAD-DEVICE 54336 tokens),
then "sgl-project#1416 STORE-TOLD ANCHOR-CLAMP rid=weg2-1-9 completed=54336 anchored=0" -- told 0, the followers
registered nothing (their STORE WITNESS unprobed), every rank prefilled from 0 (SERVED group=P
prompt_tokens=54388 cached_tokens=0). Every P read with loaded>0 in that boot clamped to anchored=0. Root:
with TOLD_PROBE_TREE_KEY off, weg2_store_told._anchor_clamp hashes origin_input_ids as UNIGRAM keys while
the fetch keys the span as the tree does (RadixKey is_bigram=tree.is_eagle, bigram on NF's MTP group) and
through the sgl-project#1442 hand-off chain -- the probe asked for keys nobody writes: the 27B #1416d class, fixed by
TK, which the nextflash row kept off.

The row now carries told_probe_tree_key, told_paced and p_twin_defer = True (TK brings TOLD_ABSOLUTE along).
TK never without PACED: sgl-project#211 (rc11b, -15 % PP0 prefill) was TK's single-phase follower busy-wait in
admission(). With PACED the only remaining wait is admission()'s bounded residual loop when a follower's
read outlasts PP0's pacing window (max(1.25 x PP0's read, told x 1 s/100k), cap 10 s). On the NF P cut PP0
carries the heaviest store share (sgl-project#706 lines: KV slots 7/12 on PP0 vs 3 and 2; GDN blob 35.95 of 58.83 MB
vs 13.07 and 9.81), so a follower's read ends inside the window; 27B metal 09271525: PACED-ADMIT 24, WAITED
0. PF (group fallback) stays off, MZ and front span inflight stay off (not released by the NF seat). An
explicitly set env still wins; profiles/nf-h91-dpr-tk.env states the same values.

Metal (NF seat, rc12u with nf-h91-dpr-tk, 19:09-19:16Z): P legs 10, 8 with a hit (the 2 cold ones first
turns), walls 2.6-8.3 s instead of ~25 s; STORE-TOLD WAITED 0, waited_max 0.62 s, ANCHOR-CLAMP 0.

Tests: test_unify_prefix_switches_profile.py updated to the new row (NF3 = TK, PACED, TW: registry,
state per profile, published form, publish writes exactly the trio, build_env differs by exactly the trio,
profile_docker facts, rank readers incl. TOLD_ABSOLUTE). Fix vs base dbeb9ff, 26 files one by one
through test27b.sh (the prefix-switch/profile suites and every weg2_store_told / p_twin_defer /
told_fallback suite): no new failure.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…D's weg2_resumable_depth (deepest held Mamba anchor)

rc12z weg2-12-61: weg2-8-53 served ct=55424 (prompt 57396), RETAIN value=False at its finish;
weg2-12-61 (57792 tokens) priced credit 55424 -> pending 2368 SHORT, D's extent 37952 ->
X refusal midstream, 72 s P detour. rc12y weg2-16-42: weg2-13-41 ct=46848, no recurrent
state after RETAIN value=False, weg2-16-42 priced 125.

Front half of the NF-agreed seam: D sends weg2_resumable_depth (meta_info on /generate,
sglext on the OpenAI wire, like weg2_prefill_s) at the leg-2 finish; the front records it
beside each presence entry (SpanLRU and TokenSpans depth_caps, entry tuples unchanged) and
credits min(credit, depth) -- the measured cached_tokens and the sgl-project#49 held prompt alike. 0
retracts. Field absent = the old credit, byte for byte. One line WEG2 PRESENCE-DEPTH-CAP
rid=... credit old -> new when it caps; counters presence_depth_capped/uncapped/absent.
Both leg-2 finish sites (stream: the finish chunk in the retained tail; body: js).

Tests test_weg2_price_anchor_59_0928.py 12 passed: 12-61 (55424 -> 37952, pending 19840 > X
5074) and 16-42 (depth 0 -> whole 46851), both also WITHOUT the field reproducing the metal
log lines (2368 / 125); held-epoch cap, inflight keeps the cap, cap trimmed with its entry,
SpanLRU, both wires + stream tail, the log line, the wiring. Red-first on 7055ec7: 10
failed / 2 passed (the two no-field controls).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht.

Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung).

Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht.

Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung).

Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…today's token spans, P-anchor presence off/on

The witness exists (PREFILL-EINBRUCH-0929 K1, a59c95a, SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, default
off) and never ran on 27B metal: 0 'WEG2 P-ANCHOR-PRESENCE' lines in every 27B boot. At the first content
of an after_p leg 2 the front records P's END anchor (page floor of P's prompt) as a presence for every
later text on that prefix.

Fixture: fixtures/front_span_49/w109290020_events.json -- 993 events of the 30-min agent load (29.09.
00:20-00:54Z, bb82fbc): flips (epoch/awake), X-EXACT prices (tokens, reused prefix), route verdicts
(X), P leg-1 and D leg-2 serves (cached/prompt/epoch/sgl-project#59 depth), leg-2 first content (via). The replay
drives TokenSpans exactly as the front does (pending with the router's epoch rule, record_presence at
the D finish, record_inflight at a d_direct first content, record_store_anchor at an after_p first
content only with the switch on); token ids are synthesised from the reused prefixes.

Result on today's code (the log predates K2, so it said 94 LONG of 182):
  OFF 58 LONG, ON 43 LONG -- 15 agent turns D-direct instead of P + 2 flips, none moved back.
  Four moved turns had a P side that held less than the credit (P leg-1 rest > X: weg2-20-21, 36-39,
  42-42, 118-130); each one's source was RUNNING on D at the price time (first content given, leg 2
  not finished), so D's radix held it. The test fails any other such move (danger direction W50).
Metal (next 27B flip boot on the RC image with the flip-levers patch): 'WEG2 P-ANCHOR-PRESENCE' > 0,
W50-REROUTE not above the off arm, tools/front_route_gap K1_FIXES -> ~0.

Tests: test_weg2_p_anchor_presence_replay_w1_0930.py 4 green; test_weg2_price_store_anchor_0929 green.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…s its sgl-project#59 depth after D's sleep leg published it

Switch SGLANG_WEG2_ENABLE_D_EPOCH_PUBLISH_PRESENCE (front only, default off). At a D->P flip done D's
sleep leg has drained its HiCache writes, published the rest (sgl-project#1470) and joined the store queue
(#1470b) before the kv pause. TokenSpans.promote_published(done_epoch): every text D served (held
credit) in an epoch before done_epoch that carries D's own sgl-project#59 weg2_resumable_depth gets that depth as
a STORE presence -- never the prompt, never past the text; no sgl-project#59 depth, no credit; the current epoch
untouched. The depth is kept in TokenSpans.published BESIDE the entry: pending() credits it only where
it lies on the new text's shared path, so the entry's lower measured anchor still credits a text that
diverges earlier (folding it into ct lost weg2-134-146 in the replay). A new D reading of the text
(a W31 refusal's small cached_tokens) pops it: the over-credit is refused, not repeated. Line
'WEG2 D-EPOCH-PUBLISH-PRESENCE', counters d_epoch_publish_presence[_tokens], then the queue reprice.

Replay (fixture re-extracted: flip begin/done, D-side 'sgl-project#59 RESUMABLE' depths from the D log; the
router's held-credit rule 'D awake AND serving'; a re-priced queued LONG reaches D only through the
short drain, i.e. while D is awake and serving) on w109290020:
  OFF 59 LONG, L1 (P-anchor presence) 38 -- 21 moved (7 by the reprice at the witness), none back
  (supersedes the 58 -> 43 of 75833a7, which had no flip state, no D depths, no reprice);
  L2 alone 56 (3 moved: weg2-15-18, 27-27, 90-99), L1+L2 38 -- L2 adds NOTHING on top of L1 here:
  the 7 DDIRECT_GAP turns were priced DURING the D->P flip already under way (no held credit, no
  witness yet); their LONG route costs no extra flip, the flip happened anyway.
Recommendation: L2 stays OFF (not in the flip-levers patch); L1 is the lever.

Tests: test_weg2_d_epoch_publish_49_0930.py 5 (witnessed depth never the prompt; no depth / current
epoch -> nothing; divergence keeps the lower anchor; DANGER: D's contradicting reading retracts ->
LONG again; front hook only at D->P and switched on). Replay file 6 (+L2 arm). test_weg2_front_x_exact:
the reprice-hook count 4 -> 5 with the new hook's guard pinned. Neighbours (40 files importing
front_tokens / X-EXACT / flip): 0 new failures vs 75833a7.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…hared path (dual layout; P reports them in meta_info)

Measured need (dual 0930 12:11, tools/front_route_gap): 11 of 18 LONG routes were repeated 40k/62k
documents whose token LCP ends BEFORE the earlier text's end anchor; P hit 36863 / 61439 / 94207 -- its
INNER anchors -- and the front credited 0 (presence_src=none, dual16 finding P2): the PX anchor rule knows
the end anchor only, and the inner anchors sit at a DISTANCE of 4096 from the last one at chunk boundaries
(weg2_anchor_step), not on a grid the front could compute.

P side: mamba_component._weg2_anchor_step_declines records every INTERVAL anchor a request keeps
(req._weg2_anchor_depths, raw token positions, rank-uniform inputs); the output streamer stamps them on the
finishing output (req_time_stats.weg2_anchor_depths, in __getstate__ only when set); tokenizer_manager puts
them into meta_info; serving_chat / serving_completions (SglExt.weg2_anchor_depths) and the Anthropic
wire (AnthropicSglExt, _sglext_of) carry them to the front.
Front: anchor_depths_of() reads them from P's leg-1 answer (Pending.leg1_anchor_depths); with the
P-anchor witness (SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, D resumed P's END anchor at its first content) AND
the new switch SGLANG_WEG2_ENABLE_INNER_ANCHOR_PRESENCE (default off) AND --dual-layout, the anchors below
the end anchor -- only the deepest MAX_STATES_PER_PATH-1, the ones P's per-path cap keeps -- go into
TokenSpans.inner beside the entry; pending() credits the deepest one <= the token LCP; a later D reading
of the text pops them. Line 'WEG2 INNER-ANCHOR-PRESENCE', counter inner_anchor_presence.
WHY DUAL ONLY: the 27B flip form releases inner anchors at P's reset (grid4096:
SGLANG_WEG2_MAMBA_INNER_ANCHOR_RELEASE) -- crediting them there over-credits (W50). The premise that a
D-direct turn reads P's published pages is K1's; the K1 metal verdict (W50-REROUTE) decides L3 too.

Tests: test_weg2_inner_anchor_presence_49_0930.py 7 -- P records its interval anchors (1024-token chunks,
anchors at 4096/8192/12288...), time_stats carry them, the Anthropic sglext carries them, the front reads
them (sglext/meta_info), the deepest kept anchor on the shared path credits (36864 on a repeat diverging
at 37500, was 0), one past the divergence credits nothing, a D reading retracts them, and the front
records them only under the dual layout with the switch on. Neighbours (246 files naming the front /
token spans / serving / tokenizer manager / mamba component / time stats): 0 new failures vs 3ec239f.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
… over P that the store would have let D serve

One front log in, no GPU/tokenizer. Per LONG route: real_rest = P leg-1 prompt - P leg-1 cached; the
source = the earlier request whose prompt equals this one's X-EXACT reused prefix; when it became a
store presence (after_p first content = the P_ANCHOR_PRESENCE witness; a D-direct source only at its
D leg-2 finish). Classes OK_LONG / K1_FIXES / INFLIGHT(after_p) / DDIRECT_GAP_served / OTHER.

Real agent trace w109290020 (29.09. 00:20-00:54Z, bb82fbc, 30 min agent load): 185 routed turns,
96 over P (+2 flips each), 34 needed P (real rest > X=4096); of the 62 others: K1_FIXES 38,
INFLIGHT 9, DDIRECT_GAP 7, OTHER 8. w109281851: 14 over P, 8 needed. Dual 0930 12:11 (P2 shape):
18 over P, 7 needed, 11 OTHER = repeated 40k/62k documents whose token LCP ends before the source's
end anchor (P hit 36863 / 61439, front credit 0, presence_src=none).

Test: test_weg2_front_route_gap_49_0930.py (2): one synthetic LONG per class, SHORT not counted.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…today's token spans, P-anchor presence off/on

The witness exists (PREFILL-EINBRUCH-0929 K1, a59c95a, SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, default
off) and never ran on 27B metal: 0 'WEG2 P-ANCHOR-PRESENCE' lines in every 27B boot. At the first content
of an after_p leg 2 the front records P's END anchor (page floor of P's prompt) as a presence for every
later text on that prefix.

Fixture: fixtures/front_span_49/w109290020_events.json -- 993 events of the 30-min agent load (29.09.
00:20-00:54Z, bb82fbc): flips (epoch/awake), X-EXACT prices (tokens, reused prefix), route verdicts
(X), P leg-1 and D leg-2 serves (cached/prompt/epoch/sgl-project#59 depth), leg-2 first content (via). The replay
drives TokenSpans exactly as the front does (pending with the router's epoch rule, record_presence at
the D finish, record_inflight at a d_direct first content, record_store_anchor at an after_p first
content only with the switch on); token ids are synthesised from the reused prefixes.

Result on today's code (the log predates K2, so it said 94 LONG of 182):
  OFF 58 LONG, ON 43 LONG -- 15 agent turns D-direct instead of P + 2 flips, none moved back.
  Four moved turns had a P side that held less than the credit (P leg-1 rest > X: weg2-20-21, 36-39,
  42-42, 118-130); each one's source was RUNNING on D at the price time (first content given, leg 2
  not finished), so D's radix held it. The test fails any other such move (danger direction W50).
Metal (next 27B flip boot on the RC image with the flip-levers patch): 'WEG2 P-ANCHOR-PRESENCE' > 0,
W50-REROUTE not above the off arm, tools/front_route_gap K1_FIXES -> ~0.

Tests: test_weg2_p_anchor_presence_replay_w1_0930.py 4 green; test_weg2_price_store_anchor_0929 green.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…s its sgl-project#59 depth after D's sleep leg published it

Switch SGLANG_WEG2_ENABLE_D_EPOCH_PUBLISH_PRESENCE (front only, default off). At a D->P flip done D's
sleep leg has drained its HiCache writes, published the rest (sgl-project#1470) and joined the store queue
(#1470b) before the kv pause. TokenSpans.promote_published(done_epoch): every text D served (held
credit) in an epoch before done_epoch that carries D's own sgl-project#59 weg2_resumable_depth gets that depth as
a STORE presence -- never the prompt, never past the text; no sgl-project#59 depth, no credit; the current epoch
untouched. The depth is kept in TokenSpans.published BESIDE the entry: pending() credits it only where
it lies on the new text's shared path, so the entry's lower measured anchor still credits a text that
diverges earlier (folding it into ct lost weg2-134-146 in the replay). A new D reading of the text
(a W31 refusal's small cached_tokens) pops it: the over-credit is refused, not repeated. Line
'WEG2 D-EPOCH-PUBLISH-PRESENCE', counters d_epoch_publish_presence[_tokens], then the queue reprice.

Replay (fixture re-extracted: flip begin/done, D-side 'sgl-project#59 RESUMABLE' depths from the D log; the
router's held-credit rule 'D awake AND serving'; a re-priced queued LONG reaches D only through the
short drain, i.e. while D is awake and serving) on w109290020:
  OFF 59 LONG, L1 (P-anchor presence) 38 -- 21 moved (7 by the reprice at the witness), none back
  (supersedes the 58 -> 43 of 75833a7, which had no flip state, no D depths, no reprice);
  L2 alone 56 (3 moved: weg2-15-18, 27-27, 90-99), L1+L2 38 -- L2 adds NOTHING on top of L1 here:
  the 7 DDIRECT_GAP turns were priced DURING the D->P flip already under way (no held credit, no
  witness yet); their LONG route costs no extra flip, the flip happened anyway.
Recommendation: L2 stays OFF (not in the flip-levers patch); L1 is the lever.

Tests: test_weg2_d_epoch_publish_49_0930.py 5 (witnessed depth never the prompt; no depth / current
epoch -> nothing; divergence keeps the lower anchor; DANGER: D's contradicting reading retracts ->
LONG again; front hook only at D->P and switched on). Replay file 6 (+L2 arm). test_weg2_front_x_exact:
the reprice-hook count 4 -> 5 with the new hook's guard pinned. Neighbours (40 files importing
front_tokens / X-EXACT / flip): 0 new failures vs 75833a7.


(cherry picked from commit 3ec239f, desk/27b-front-span-49-0930; environ.py: beside the RPC-STALL-WATCHDOG entry)
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 30, 2026
…hared path (dual layout; P reports them in meta_info)

Measured need (dual 0930 12:11, tools/front_route_gap): 11 of 18 LONG routes were repeated 40k/62k
documents whose token LCP ends BEFORE the earlier text's end anchor; P hit 36863 / 61439 / 94207 -- its
INNER anchors -- and the front credited 0 (presence_src=none, dual16 finding P2): the PX anchor rule knows
the end anchor only, and the inner anchors sit at a DISTANCE of 4096 from the last one at chunk boundaries
(weg2_anchor_step), not on a grid the front could compute.

P side: mamba_component._weg2_anchor_step_declines records every INTERVAL anchor a request keeps
(req._weg2_anchor_depths, raw token positions, rank-uniform inputs); the output streamer stamps them on the
finishing output (req_time_stats.weg2_anchor_depths, in __getstate__ only when set); tokenizer_manager puts
them into meta_info; serving_chat / serving_completions (SglExt.weg2_anchor_depths) and the Anthropic
wire (AnthropicSglExt, _sglext_of) carry them to the front.
Front: anchor_depths_of() reads them from P's leg-1 answer (Pending.leg1_anchor_depths); with the
P-anchor witness (SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, D resumed P's END anchor at its first content) AND
the new switch SGLANG_WEG2_ENABLE_INNER_ANCHOR_PRESENCE (default off) AND --dual-layout, the anchors below
the end anchor -- only the deepest MAX_STATES_PER_PATH-1, the ones P's per-path cap keeps -- go into
TokenSpans.inner beside the entry; pending() credits the deepest one <= the token LCP; a later D reading
of the text pops them. Line 'WEG2 INNER-ANCHOR-PRESENCE', counter inner_anchor_presence.
WHY DUAL ONLY: the 27B flip form releases inner anchors at P's reset (grid4096:
SGLANG_WEG2_MAMBA_INNER_ANCHOR_RELEASE) -- crediting them there over-credits (W50). The premise that a
D-direct turn reads P's published pages is K1's; the K1 metal verdict (W50-REROUTE) decides L3 too.

Tests: test_weg2_inner_anchor_presence_49_0930.py 7 -- P records its interval anchors (1024-token chunks,
anchors at 4096/8192/12288...), time_stats carry them, the Anthropic sglext carries them, the front reads
them (sglext/meta_info), the deepest kept anchor on the shared path credits (36864 on a repeat diverging
at 37500, was 0), one past the divergence credits nothing, a D reading retracts them, and the front
records them only under the dual layout with the switch on. Neighbours (246 files naming the front /
token spans / serving / tokenizer manager / mamba component / time stats): 0 new failures vs 3ec239f.


(cherry picked from commit 93c2691, desk/27b-front-span-49-0930; front.leg1: beside the dual20 leg1_aborted line)
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 2, 2026
…e D-Leg-2 kreditiert ihren End-Anker, auch ueber den D-Schlaf

NF boot dkrnfint4bar1dauer10020634 (5b46b88), nextflash row agent_span=0: a FINISHED D leg 2
was credited only with its admission hit (record_presence dropped prompt_tokens/held_epoch).
weg2-26-44 served 06:42:29,440 pt=53748 ct=50688 depth 53760; 44 ms later weg2-26-45
(SESSION-PREFIX common=53748) 'X-EXACT-PRICE pending=4582 credit=50688' > X=3647 -> LONG,
P prefilled 1510 on a 53760 hit. weg2-24-40: credit 45824 (24-39's ct), 4339 > 3602, P 2291.

(1) Same epoch: the nextflash row runs sgl-project#49 (form.py agent_span=True; released by the NF seat).
Its premises hold on NF: the Flash-Next template renders <tools> first (chat_template.jinja
57-67), D sends weg2_resumable_depth on every finish (sgl-project#59 RESUMABLE mode=form-a-dcp-min).
A served leg then holds min(prompt, D's depth) for its epoch (src=d_served_epoch): 26-45
credit 53696 pending 1574, 24-40 47808 / 2355 -- both SHORT.
(2) Across epochs: the sgl-project#49 epoch guard protects the radix-only tokens past the anchor. With
a sgl-project#59 depth the credit is min(prompt, depth, page floor) and survives D's sleep as a store
load (src=d_served_anchor): in this boot 14 of 14 follow-ups after a D sleep read >= that
anchor (26-45 P hit 53760 >= 53696, 27-46 55232, 64-104 on D 88336 >= 85888), 0 ANCHOR-LOST.
The load is not priced: the LONG path loads the same prefix (P store read, D LOAD-DEVICE
after the hand-off), X is a prefill bound. Without a depth the epoch guard stands. ANCHOR-LOST
now also matches D's unclamped depth (TokenSpans.raw_depths; the K2 clamp had hidden it).
(3) d_inflight (fceeda8) only for after_p legs (P's END-ANCHOR, realised): a prompt D
prefilled itself anchors on its prefill track, 8 of 54 in-flight 'sgl-project#1469 RETAIN value=True'
were below the page floor (weg2-4-7 19509 -> 17728); its finish reading covers it.

Replay of the boot (25 LONG with real P prefill <= X): finished-same-epoch 9, cross-epoch 5,
in-flight after_p 4 (attributed in that order; alone 9 / 8 / 11) = 18 SHORT.

Tests: test_weg2_x_credit_finished_1002.py (11; 8 red on base: 26-45, 24-40, row, cross x3,
no-depth guard, ANCHOR-LOST raw depth). Pins updated: agent_span profile rows (nextflash on),
sgl-project#59 held-cap pin (outside the epoch 640 d_served_anchor), H61/H61b read the booked entry
instead of SpanLRU.span_tokens (a chars/3 reader under sgl-project#49). Front-importing suites A+B
(171 files) under the nextflash form: base 8+23 failed, fix 8+23 failed, 0 new reds
(the 21 x_credit tests red on base only); no form: 507 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 20, 2026
FLLIPER_MOE_MARLIN_C_SENTINEL=2 scans GEMM1 / ACT / GEMM2 rows for non-finite values. In the EP
form (expert map, or a pool EM wave's filtered lanes) a -1 lane's GEMM1 / ACT rows are never
written -- the GEMMs skip their blocks. _live_lanes_only folds a stage's bad (token, k) rows to
tokens over LIVE lanes only (topk_ids >= 0) when is_ep; outside the EP form unchanged. With the
sentinel on the arena is pre-filled with a finite value, so today no false alarm was reachable;
this removes the dependency. The level-1 C-probe reads the output after GEMM2, whose cache is
zeroed in the EP form: unaffected.
Tests (serial, MemoryMax=2G, CUDA_VISIBLE_DEVICES=""): W2 + S51 + Marlin/sgl-project#49 suites 117 passed,
1 skipped; mutant (no exclusion) red.
TianHao65 added a commit to TianHao65/sglang that referenced this pull request Nov 3, 2026
Port the packaged workload drivers from PR sgl-project#13 onto sammysun0711/sglang:ci/qwen4-unified, record strict draft counters and weighted acceptance length, and retain the original GSM8K accuracy entry point. Add dataset validation, CPU regression checks, and a linked usage guide for issue sgl-project#49.
TianHao65 added a commit to TianHao65/sglang that referenced this pull request Nov 3, 2026
Port the DFlash acceptance changes developed on TianHao65/sglang:ci/qwen4-unified to the apinge/sglang qwen4_v0.5.21 target. Record strict draft counters and weighted acceptance length, preserve the original GSM8K accuracy entry point, and add dataset validation, CPU checks, and documentation for issue sgl-project#49.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants