Repository navigation
Support stream=True in v1/completions - #49
Merged
Merged
Conversation
comaniac
force-pushed
the
cody/openai-completion
branch
from
January 18, 2024 22:58
6d9fbec to
a6acdf3
Compare
Collaborator
|
This looks good. Let me take this over and make the frontend compatible with your new changes |
Contributor
Author
|
Sure. Feel free push necessary changes, and I'll rebase the chat completion API once this PR is merged. |
5 tasks done
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
chunyuan-w
pushed a commit
to chunyuan-w/sglang
that referenced
this pull request
Apr 15, 2025
Add fp8 for shared_expert
NorthmanPKU
pushed a commit
to NorthmanPKU/sglang
that referenced
this pull request
May 16, 2025
* Update cython_setup.py * Update __init__.py
pi314ever
added a commit
to pi314ever/sglang
that referenced
this pull request
Aug 14, 2025
* Fix sglang llama4 rotary embedding and introduce mm inputs to hpu forward batch Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Enable working llama4 image processing, accuracy issue present Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Fix vision rotary embedding forward Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Bring in models to use native sglang classes Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Revert "Bring in models to use native sglang classes" This reverts commit d61f0e7b68e52591b853ace7b79179a6b22940ea. * Add disable cuda graph to disable hpu graphs Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Convert ndarray to tensors too Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Fix mm accessing Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Multimodal graph mode workaround Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * remove print statement * ruff fix autofixable Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Fix syntax issue Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Fix rot emb no-vllm logic Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Fix is_cpu references in mllama4 Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Minor fixes broken from rebase Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * isort and black formatting Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * Allow batched multimodal Signed-off-by: Daniel Huang <daniel1.huang@intel.com> * precommit format Signed-off-by: Daniel Huang <daniel1.huang@intel.com> --------- Signed-off-by: Daniel Huang <daniel1.huang@intel.com> Co-authored-by: Yang Wang <y3wang@habana.ai>
3 of 5 tasks
intervitens
pushed a commit
to intervitens/sglang
that referenced
this pull request
Aug 31, 2025
key4ng
pushed a commit
to key4ng/sglang
that referenced
this pull request
Nov 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
wisclmy0611
pushed a commit
that referenced
this pull request
Apr 7, 2026
* docs: improve information density * tighten information density * fixed broken links * docs: sync LMSYS SGLang blog cards * fix sidebar * feat: docs restructure --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 16, 2026
Two GGUF-specific bugs, one file. 1. (sgl-project#49) GGUF-MTP draft path over-reserved KV 5x: max_total_num_tokens 100032 instead of ~470k. Root cause: _config_draft_model() sets num_nextn_predict_layers=1 on hf_config, but for Qwen3.5 that is the multimodal WRAPPER config while _derive_model_shapes() reads the field from hf_text_config (inner text config, which only carries mtp_num_hidden_layers) -> draft reports None -> model_runner falls back to the full target depth (num_hidden_layers=64) for draft KV sizing -> spec KV cell scaled by (1 + 64/16) = 5x. FP8 never hits this (embedded NEXTN head, no --speculative-draft-model-path). Fix: fall back to hf_config when the text config lacks the field (fires only when a draft rewrite set it; targets stay None). Measured: GGUF-MTP max_total 100032 -> 470912 (4.71x, per-rank [242698, 125088, 125209]; FP8 parity 475200). 2. (sgl-project#52) GGUF-only early-boot corruption: on a fresh boot, prompts >=64 tokens corrupted from ~token 2 until one /flush_cache healed permanently. M16's radix framing was wrong: victims had cached-token: 0, and order-swap showed victim selection is prompt length >=64 (mamba chunk grid), not request ordinal. Actual chain: sglang's server warmup sends a real image request whenever the model advertises image understanding -> the GGUF checkpoint has ZERO vision tensors (llama.cpp keeps vision in a separate mmproj file; the loader maps text+NEXTN only) -> warmup runs an UNINITIALIZED vision tower -> NaN image features contaminate that request's GDN/mamba states -> NaN residue in freed/recycled mamba slots -> every >=64-token prefill (track path) picks it up. FP8 runs the same path with real vision weights (finite residue, invisible). /flush_cache heals via mamba_pool.reset_state() (KV zeroing exonerated via SGLANG_FLUSH_ZERO_KV=0 control; --skip-server-warmup boot fully clean, isolating the source). Fix: force enable_multimodal=False for GGUF checkpoints (a checkpoint without vision weights is not an image-understanding model) -- warmup and API take the text path; image requests can no longer NaN-poison mid-serving either. FP8 multimodal untouched. Validation: - GGUF no-MTP: max_total 840896 unchanged; worst-case request order (74-tok victim first) all clean, NO flush needed. - GGUF MTP fresh boot: max_total 470912; request sequence clean without flush; Paris + 17x3 pass; 1024-tok greedy long-form clean; conc-4 4/4 clean; accept_len 2.48-3.44 (matches M17 content dependence). - FP8-MTP regression: max_total exactly 475200, coherence pass, warmup still the image-chat path -> no cross-quant damage. Open: real GGUF image support would need mmproj loading (rejected cleanly for now); latent deferred-clear gap on recycled mamba track slots is harmless with finite residue but would resurface with any future NaN source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 16, 2026
GGUF checkpoints follow the llama.cpp convention of storing the vision encoder in a separate mmproj*.gguf next to the text backbone. Until now this fork skipped vision tensors entirely and (since 770fbb4) forced GGUF to text-only. Port the vLLM-fork plugin's mmproj support so a qwen35-family GGUF with an mmproj file keeps multimodal ENABLED and loads a fully initialized vision tower; GGUF without mmproj keeps the text-only fallback unchanged. - gguf_qwen35.py (+168): detect_gguf_multimodal() (plugin glob rule: mmproj.gguf / mmproj-*.gguf / *mmproj*.gguf next to the backbone), clip->HF vision name maps ported from the plugin's qwen35 weights adapter, build_vision_name_map() + vision_weights_iterator() -- dense mmproj reader (gguf-py returns BF16 as raw uint8 -> view(bfloat16); F16/F32 shaped; llama.cpp's temporal Conv3d split v.patch_embd.weight / .weight.1 re-stacked at dim 2; unmapped mmproj tensor = hard error). Draft (NEXTN) configs never load vision. - loader.py (+15): GGUFModelLoader chains the vision iterator after transform_stream() iff mmproj_path is present. - model_config.py: inside the GGUF multimodal gate, a qwen35-arch ForConditionalGeneration with detected mmproj keeps multimodal enabled (logged); every other case takes the exact previous text-only disable path. Sharding: NO replication -- mmproj names are normalized to exactly the HF names the FP8 checkpoint loads (fused model.visual.*attn.qkv), so GGUF vision inherits the fork's proven uneven-TP whole-head split (6/5/5 over TP=3 auto). Offline proof: 333 emitted tensors == FP8's 333 visual tensors (names+shapes), values BIT-EXACT vs the FP8 vision weights (maxdiff 0.0). Validation (unsloth Qwen3.6-27B-MTP-GGUF + mmproj-BF16.gguf, tp=3 --rank-tp-ratio auto, weighted DCP, fp8 KV, ctx 32768): - eager no-MTP: max_total 840896 unchanged; VLM warmup runs the real vision path; test image (red circle, "STOP") described correctly: shape circle, bright red, text STOP in white capitals; Paris/17x3/ >=64-token prompts clean WITHOUT flush (old NaN victim class dead). - MTP + CUDA graphs: max_total 470912 (no collapse of the sgl-project#49 fix); image correct; 1024-token long-form clean; conc-4 (1 image + 3 text) all clean; accept_len 2.08-3.40 with graphs on. - FP8-MTP graphs regression: max_total exactly 475200, coherence clean, zero GGUF-multimodal log lines -> non-GGUF path untouched. - GGUF without mmproj: identical text-only log + is_multimodal=False (offline ModelConfig check). Open: huihui/heretic f16 mmproj variants verified offline (detection + decode) but not live-booted; vision runs eager (same as FP8 today). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 22, 2026
Two GGUF-specific bugs, one file. 1. (sgl-project#49) GGUF-MTP draft path over-reserved KV 5x: max_total_num_tokens 100032 instead of ~470k. Root cause: _config_draft_model() sets num_nextn_predict_layers=1 on hf_config, but for Qwen3.5 that is the multimodal WRAPPER config while _derive_model_shapes() reads the field from hf_text_config (inner text config, which only carries mtp_num_hidden_layers) -> draft reports None -> model_runner falls back to the full target depth (num_hidden_layers=64) for draft KV sizing -> spec KV cell scaled by (1 + 64/16) = 5x. FP8 never hits this (embedded NEXTN head, no --speculative-draft-model-path). Fix: fall back to hf_config when the text config lacks the field (fires only when a draft rewrite set it; targets stay None). Measured: GGUF-MTP max_total 100032 -> 470912 (4.71x, per-rank [242698, 125088, 125209]; FP8 parity 475200). 2. (sgl-project#52) GGUF-only early-boot corruption: on a fresh boot, prompts >=64 tokens corrupted from ~token 2 until one /flush_cache healed permanently. M16's radix framing was wrong: victims had cached-token: 0, and order-swap showed victim selection is prompt length >=64 (mamba chunk grid), not request ordinal. Actual chain: sglang's server warmup sends a real image request whenever the model advertises image understanding -> the GGUF checkpoint has ZERO vision tensors (llama.cpp keeps vision in a separate mmproj file; the loader maps text+NEXTN only) -> warmup runs an UNINITIALIZED vision tower -> NaN image features contaminate that request's GDN/mamba states -> NaN residue in freed/recycled mamba slots -> every >=64-token prefill (track path) picks it up. FP8 runs the same path with real vision weights (finite residue, invisible). /flush_cache heals via mamba_pool.reset_state() (KV zeroing exonerated via SGLANG_FLUSH_ZERO_KV=0 control; --skip-server-warmup boot fully clean, isolating the source). Fix: force enable_multimodal=False for GGUF checkpoints (a checkpoint without vision weights is not an image-understanding model) -- warmup and API take the text path; image requests can no longer NaN-poison mid-serving either. FP8 multimodal untouched. Validation: - GGUF no-MTP: max_total 840896 unchanged; worst-case request order (74-tok victim first) all clean, NO flush needed. - GGUF MTP fresh boot: max_total 470912; request sequence clean without flush; Paris + 17x3 pass; 1024-tok greedy long-form clean; conc-4 4/4 clean; accept_len 2.48-3.44 (matches M17 content dependence). - FP8-MTP regression: max_total exactly 475200, coherence pass, warmup still the image-chat path -> no cross-quant damage. Open: real GGUF image support would need mmproj loading (rejected cleanly for now); latent deferred-clear gap on recycled mamba track slots is harmless with finite residue but would resurface with any future NaN source.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Jul 22, 2026
GGUF checkpoints follow the llama.cpp convention of storing the vision encoder in a separate mmproj*.gguf next to the text backbone. Until now this fork skipped vision tensors entirely and (since 39176df) forced GGUF to text-only. Port the vLLM-fork plugin's mmproj support so a qwen35-family GGUF with an mmproj file keeps multimodal ENABLED and loads a fully initialized vision tower; GGUF without mmproj keeps the text-only fallback unchanged. - gguf_qwen35.py (+168): detect_gguf_multimodal() (plugin glob rule: mmproj.gguf / mmproj-*.gguf / *mmproj*.gguf next to the backbone), clip->HF vision name maps ported from the plugin's qwen35 weights adapter, build_vision_name_map() + vision_weights_iterator() -- dense mmproj reader (gguf-py returns BF16 as raw uint8 -> view(bfloat16); F16/F32 shaped; llama.cpp's temporal Conv3d split v.patch_embd.weight / .weight.1 re-stacked at dim 2; unmapped mmproj tensor = hard error). Draft (NEXTN) configs never load vision. - loader.py (+15): GGUFModelLoader chains the vision iterator after transform_stream() iff mmproj_path is present. - model_config.py: inside the GGUF multimodal gate, a qwen35-arch ForConditionalGeneration with detected mmproj keeps multimodal enabled (logged); every other case takes the exact previous text-only disable path. Sharding: NO replication -- mmproj names are normalized to exactly the HF names the FP8 checkpoint loads (fused model.visual.*attn.qkv), so GGUF vision inherits the fork's proven uneven-TP whole-head split (6/5/5 over TP=3 auto). Offline proof: 333 emitted tensors == FP8's 333 visual tensors (names+shapes), values BIT-EXACT vs the FP8 vision weights (maxdiff 0.0). Validation (unsloth Qwen3.6-27B-MTP-GGUF + mmproj-BF16.gguf, tp=3 --rank-tp-ratio auto, weighted DCP, fp8 KV, ctx 32768): - eager no-MTP: max_total 840896 unchanged; VLM warmup runs the real vision path; test image (red circle, "STOP") described correctly: shape circle, bright red, text STOP in white capitals; Paris/17x3/ >=64-token prompts clean WITHOUT flush (old NaN victim class dead). - MTP + CUDA graphs: max_total 470912 (no collapse of the sgl-project#49 fix); image correct; 1024-token long-form clean; conc-4 (1 image + 3 text) all clean; accept_len 2.08-3.40 with graphs on. - FP8-MTP graphs regression: max_total exactly 475200, coherence clean, zero GGUF-multimodal log lines -> non-GGUF path untouched. - GGUF without mmproj: identical text-only log + is_multimodal=False (offline ModelConfig check). Open: huihui/heretic f16 mmproj variants verified offline (detection + decode) but not live-booted; vision runs eager (same as FP8 today).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 19, 2026
…inite Task sgl-project#49 (19.09.): three 262k boots answered '!!!' x768 (token 0 = NaN logits) -- fn7n (no-enable-multimodal + INT8 mixers), fn7u (the flag alone), fn7w (the INT8 mixers alone) -- while fn7l/fn7v/fn7t matched the needle at 259k; short prompts are always right. Whether that is two causes or one non-deterministic edge, the log so far cannot say: nothing reports where the numbers die. With SGLANG_NAN_GUARD=1 every Qwen4Exp layer checks its MoE output once per forward (skipped under capture) and logs the FIRST non-finite hit per layer with forward mode, rows and context depth; later hits are counted. Off by default, zero cost. Tests: tests/moe_offload/test_nan_guard.py (1); suite tests/moe_offload 551 passed / 43 failed (baseline 43).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 20, 2026
… SGLANG_PDL_ON_SM12=1 fn8c12: the Triton fused-gate router, launched with PDL and carrying its gdc_wait, still turned finite gate logits into whole-token NaN topk_weights on the RTX 5090 (ROUTER-ORIGIN ENTERS-AT-TOPK-WEIGHTS bad_logit_rows=0 bad_weight_tokens=48 pairs_per_bad_token=10); the same kernel isolated from any predecessor is clean, and with the Triton router off (fn8c10b) the boot had no event in 4 needles. On sm_120 the griddepcontrol barrier does not order the dependent's read -- the class CUTLASS documents for compute_120 builds. is_arch_support_pdl() now answers False for major >= 12 (one INFO line per process), which covers all eight launch_pdl sites; the opt-in keeps the A/B spellable. Gate: tests/moe_offload 43 failed (baseline set), no new failure.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 20, 2026
… SGLANG_PDL_ON_SM12=1 fn8c12: the Triton fused-gate router, launched with PDL and carrying its gdc_wait, still turned finite gate logits into whole-token NaN topk_weights on the RTX 5090 (ROUTER-ORIGIN ENTERS-AT-TOPK-WEIGHTS bad_logit_rows=0 bad_weight_tokens=48 pairs_per_bad_token=10); the same kernel isolated from any predecessor is clean, and with the Triton router off (fn8c10b) the boot had no event in 4 needles. On sm_120 the griddepcontrol barrier does not order the dependent's read -- the class CUTLASS documents for compute_120 builds. is_arch_support_pdl() now answers False for major >= 12 (one INFO line per process), which covers all eight launch_pdl sites; the opt-in keeps the A/B spellable. Gate: tests/moe_offload 43 failed (baseline set), no new failure. (cherry picked from commit 0d33570)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 25, 2026
… -- proof obligations pinned + window stress test Frage (Operator 24.09. ~21:10Z, NF-Sitz bestaetigt dieselben Zeilen): das dichte Template jit_kernel/csrc/gemm/marlin/marlin_template.h legt sh_red auf sh_b (:638-639) und schreibt im Epilog sh_red (thread_block_reduce :1526) direkt nach cp_async_wait<0>() (:1513) ohne Blockbarriere. Braucht es die Barriere, die NF im MoE-Zwilling eingebaut hat (b8c451a, SGLANG_MARLIN_EPILOGUE_SYNC)? Befund (Desk): nein, kein LEBENDES Byte ist erreichbar. Die letzte Blockbarriere einer Slice ist wait_for_stage() bei k == b_sh_wr_iters-2; die Operanden beider letzter matmul() sind VOR ihr geladen (Schleifenform fetch(k+1) -> Barriere -> matmul(k)), matmul liest nur Register. Der einzige sh_b-Lesezugriff danach ist der Prefetch fetch_to_registers(b_sh_wr_iters, (pipe+1)%stages) fuer eine Iteration, die es am Slice-Ende nicht gibt: break -> Epilog -> start_pipes() laedt frag[0] hinter einer Barriere neu (oder Kernelende). sh_g_idx/sh_zp/sh_s/ sh_a liegen hinter max(sh_red, sh_b); cp.async ist beim Epilog keiner mehr unterwegs (pred slice_iters >= stages, wait<stages-2>); write_result endet und barrier_release beginnt mit __syncthreads, die naechste Slice kann keinen sh_red-Leser ueberholen. Uebrig bleibt ein FORMALES Data-Race auf einem toten Load (racecheck wuerde es melden), ohne Wirkung auf Ergebnisse. Die MoE-Schleife hat dieselbe Form; NFs Metall-A/B fn8c2 (Barriere AN, 20.09.) zeigte die sgl-project#49-NaNs trotzdem, Wurzel war PDL auf sm_120 (0d33570; 27B: f24c7fd). Das dichte Template und gptq_marlin.cuh sind auf NF (f398b48) und 27B-Linie byte-gleich; dieser Commit aendert keinen Kernel (kein JIT-Neubau je Rang). Wer faehrt das dichte Template: 27B-Linie nur der DFlash2-W8-Draft auf D (CompressedTensorsWNA16 -> apply_gptq_marlin_linear -> JIT gptq_marlin), TP0 sm_120 + TP1/TP2 sm_86 (xsn436 D: "dense Marlin linears" 0.88/0.41/0.41 GiB, Marlin-bf16-Hinweis TP1/TP2); P baut den Draft, rechnet ihn nie (DFLASH-PRODUCE-ON-P off). Ziel gdncov ist W8A8 -> int8_scaled_mm, kein Marlin. Tests: - test_marlin_epilogue_alias_0924.py (hermetisch, liest nur den Template-Text): die sieben Beweisbedingungen + "sh_red nur im Epilog". 8 passed. Mutanten- Probe (6 Mutationen der Vorlage: Load hinter die Barriere, sh_b-Lesen in matmul, Tail-Barriere von write_result weg, sh_g_idx in den Alias, unpraedizierter Fetch, kein Reload in start_pipes): jede von genau ihrem Test gefangen, Original gruen. - test_marlin_epilogue_alias_gpu_0924.py (Fenster, TEST27B_GPU=1, eine Karte): haemmert den UNVERAENDERTEN Kernel an den Draft-Shapes je D-Rang (58,25,25) + 2 small-N/large-K, M 1/8/16/256/2048, Arme prod/fp32/bf16red/(atomic ab sm_90), 20 Wiederholungen; Gewichte 128+-4 (ein Alias-Treffer waere ~1.6 rms gross), Referenz = Marlins eigene bf16-Dequant. Rot bei nicht-finit oder Grenze; bitweise Lauf-zu-Lauf-Abweichung innerhalb der Grenze wird nur berichtet (sgl-project#190-Klasse, 4de61b5). JIT nur lesend: der Vorcheck faehrt den ECHTEN load_jit-Pfad mit ersetzten Schreibern (heal_entry/building_marker/ purge_entry/_selfheal_jit_cache_once) und Sonden-load_module; Desk-Probe mit erzwungener Arch: 12.0 und 8.6 -> LOADS ~/.cache/tvm-ffi/...gptq_marlin_bf16_t_ d6053cd6e445c563 (+ repack 7b12d3073e80e78d), gemeinsamer Cache unveraendert (mtime, 338 Eintraege); ohne GPU skip. GPU-Lauf: offen, Operator-Fenster. (cherry picked from commit 38c008c)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…cks, chronologisch) Grundlage: Präsenz-Scan aller 230 27B-Commits seit 76f8deb gegen diesen Baum (Stichprobe der hinzugefügten Zeilen je Commit); die 94 fehlenden minus die bewusst anders gewählten Formen (76e87ac/4ae11ababd -> S3 form.calibration_identity; 479f6ec/d7f588e017/d0fba8955f/34892e3017 -> S2 NF-Formen; 7f81f09/3c14481318 -> S4/S7a; 3dbb790 line_gate_27b (Werkzeug, Schritt 9); 8604d13 W100-by-name (Nutzer: bleibt aus); 6545e2c flashinfer-Pin in pyproject (Image-Frage, nicht Baum)). Liste: 92bbccb eb5d044 829ebd0 431fcbc ef4d11f 6816062 a233e50 2cc593c f0c8451 87cc4fb c529777 31f2dbe c50085a 3301a96 036b368 e1d1fe9 03c68af 6dddc2e 06932b5 87389c4 58a7490 f85ac55 fc64aa5 5aa24dd 97c0e9a 159333c d9f1532 f3c685b 8550655 e50fb59 db2c2ef f09dc0d c255e10 51b810e 28a55a2 34965fc ff3d9cc 340a018 bee5e10 67b6352 fdade85 ed6630d f1c9a43 b434831 517f26d 0b6b60b a40837f 644de86 aff2b7c 197b701 856024b 238512a 9738626 b857a22 1f8c24d d294b3e 810239d b429dfd e714c95 9efd974 3d63e0a d3cfcf3 fee6134 7985b56 49a14e9 fc45706 19c720e 5306bee 6f1235a c98eaa3 93bc802 328349e f9fb3a2 572af73 94fa8b4 d342caa 2fd7d7e 3ebbb96 871d55f 78c2f16 3babf51 196f6a8 e70af54 22eccfc Inhalt: Upstream-Ports (sgl-project#33758 sgl-project#37818 sgl-project#36738 sgl-project#33459/sgl-project#30096 sgl-project#34446 sgl-project#36267 sgl-project#33778 sgl-project#34859 sgl-project#36415 sgl-project#35255/sgl-project#36638 sgl-project#39858/sgl-project#40259 sgl-project#31417 sgl-project#34892 sgl-project#32225 sgl-project#30832/sgl-project#36626 sgl-project#39574 sgl-project#29579 sgl-project#31468 sgl-project#32575 sgl-project#31648); xsn409/410-Wake-Verdikte; Vision-Linie V1-V3b + xsn438 (SGLANG_WEG2_VISION_FLIP_URGENT); D-Planer L6 (159333c); DFLASH-Window-Pool sync-frei, PLAN_SYNC_FREE, D-Kollektive (vocab-argmax, a2a-Merge, deferred rebuild), #DGAP/D_DEFER_SEQ_LENS_CPU; Mamba-Anker Raster 4096 + Per-Path-Cap + Inner-Release; P-TRIM (--p-trim-end-anchor); FP8 uniform Marlin; ModelOpt/NVFP4 RadixArk; GGUF G1-G6 + F1/F2; native-mixed sgl-project#38 (sm_8x W4A8, sm_12x CUTLASS/W4A16); RC1-Capture-Set; sgl-project#49 Agent-Turns; dynchunk (--p-chunk-policy, --p-chunk-dynamic-min-tokens). Auflösungen (Gabel -> Form, Grund): - L6 d_operating_point_rows: 27B (d) "Token-Vektor auf jeder Position aus der Kapazität" nur bei TP-symmetrischem D (Profil d_layout paged_dcp); sonst NF-sgl-project#1293-Pin + NF-Anker- Klausel. mamba_ssm_dtype aus EARLY_READ_FACTS nur bei Profil early_read_flags. Overhead-Kalibrierung liest mit form.CalibrationIdentity statt LineIdentity. - RC1 Capture-Set: neuer RecordKey-Term d_capture_set (qwen27b), Leser CalibrationIdentity.d_max_running_requests; nextflash unverändert. - URC Carrier-Hold: 27B _weg2_carrier_hold entfällt (S2 NF-Rotation), Inner-Release und Per-Path-Cap bleiben (Env, Default aus; 27b.env setzt sie). - scheduler_pp_mixin/overlap_utils/batch_result_processor: NF H49/H58 und 27B #PGAP/#DGAP komponiert (beide Instrumente getrennt schaltbar). - schedule_policy: P-TRIM-Kurzschluss vor NF H63-Fold/QSA-Korn; sgl-project#36415 Hoist + NF computed_input_len. - gdn_backend sgl-project#33778: 27B-strided-Verify; NF-Ring flacht beide Layouts ab. - flashinfer_backend: RC9-Datei + NF-Form-A-Waiver (27B-intern mehrfach gegabelt). - checkpoint_census: GGUF-Leser + NF exclude_segments (PLE) in einer Aggregation. - xchg_manifest: FLAT_SEGMENTS in beiden (dst/src) NF-Breitenbedingungen ausgenommen. - vram_peak_window: NF-Kumulativ-Peak liest über den 27B-Fast-Read. - FP8: 8c86eb8-Rest nachgezogen (private Workspace-Registry entfernt, wie RC9). - argv_d: vision= an allen drei Aufrufstellen (inkl. NF --d-only). - census_checkpoint_decision (W161) jetzt für beide Profile aktiv. Gates: py_compile aller geänderten Dateien; ruff F821/F811 ohne neue Funde gegenüber dem Vorgänger (PendingSeqLensCpu ist String-Annotation wie in RC9); dup_defs_gate 0 neu. Tests: 73 portierte Testdateien, Lauf nach dem Ruhefenster (Boot aktiv).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…: in-flight span credit only, sgl-project#49 itself was already in (S7c) Operator 26.09.: FS ported sgl-project#49 (27B 196f6a8) onto c1a1a00 behind SGLANG_WEG2_FRONT_SPAN_49, because sgl-project#49 was missing there. The unified tree has had sgl-project#49 unswitched since S7c (9266605, pick list includes 196f6a8; SpanLRU entries (text, ct, pt, held), record_presence(prompt_tokens, held_epoch), tools first, test_weg2_front_span_agent_49.py present). So only FS's in-flight half is taken here, with no second copy of sgl-project#49 and no _49 switch: - environ: SGLANG_WEG2_FRONT_SPAN_INFLIGHT = EnvBool(False) - front: front_span_inflight(), SpanLRU.record_inflight, stream_has_content, and the leg-2 hook: first content event -> credit the text for this epoch (switch on); WEG2 LEG2-FIRST-CONTENT rid= epoch= via= leg2_ms= instrument (always on, routing-neutral). The code is verbatim from FS. - test_weg2_front_span_inflight_fs.py: FS's file without the four _49-switch/off-path tests (tools-last, pre-port arithmetic, call-site source pins, inflight-without-_49), plus test_switch_default_off, which pins that no front_span_49 exists. Gates: dup_defs_gate vs 27b-rc9-103712cdb2 + nf-rc2.1l-d1c7094ba6: 0 new. test27b.sh: front_span_inflight_fs + front_span_agent_49 + store_consistency_1324: 43 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…NG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE, default off) Claude Code 2.1.280 sends mid-conversation role:"system" messages (api_system: tool additions/removals, output_config). With the Qwen3.8 template (head-only system turn) the Anthropic adapter hoisted them into the head system turn, so the first turn that carries one re-renders every token behind the system text. Measured on D: radix sgl-project#1420 WALK-STOP and the arena key chain (sgl-project#1439 leading_complete) both stop at the end of the system turn -- depth 3876 (dkr27bnvfp4bar1agent09252328, weg2-14-16/-17) and 4655 (dkr27bbar1final09260145, weg2-12-12), stored (13,248046)=".<|im_end|>", new (13,198)=".\n" (the "\n".join of the hoist). The sgl-project#904 census then reads MambaComponent:absent at that depth, X-GATE W31, X-REQUEUE, one P epoch. The front's request_text renders the same message in place and credits the whole previous turn (presence_span 19335/20168), so it routes SHORT. Switch on: non-leading inline system messages become a user turn at their position, wrapped in <system-reminder>; the leading run is still merged into the head. Must be set identically on P and D (handoff keys are P's tokenization). Off = byte-identical to before. Tests (test27b.sh, CPU): test_inline_system_in_place.py 5 passed (off pins the break at ".\n", on pins turn N as a rendered prefix of turn N+1); anthropic/test_serving.py + test_anthropic_557.py + test_anthropic_rid_h90.py + weg2/test_weg2_messages_wire_rc7_0925.py: 100 passed together. UNIFY (UN6, Operator 26.09.): pick of desk/27b-mambaend-0926 (2303948, MZ, base 0644c6f) after the sgl-project#49-rest port, clean. dup_defs_gate: 0 new. test27b.sh: test_inline_system_in_place + anthropic/test_serving + test_anthropic_557 + test_anthropic_rid_h90 91 passed, 9 subtests.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…le field -- qwen27b on, nextflash off (operator decision 26.09.; NF P49 c1988ff) sgl-project#49 (27B 196f6a8) ran unswitched in the unified tree since S7c (9266605), i.e. for nextflash too -- invisible in the byte-identical NF argv. NF gates it behind SGLANG_WEG2_ENABLE_AGENT_SPAN (P49 c1988ff, default off = rc2.1l + H100). Taken here with that commit's front.py and tests (conflicts line by line: environ next to the FS/S7 entries, front's request_text signature next to front_span_inflight), and the default made a MODEL PROFILE field (ModelProfile.agent_span -> PROFILE_SWITCH_DEFAULTS): qwen27b on (the 27B behaviour since RC9, unchanged), nextflash off until the NF seat releases it with a boot tag, off without a form (the NF code default). An explicit env value wins. Off = P49's form: request_text renders tools last, record_presence drops prompt_tokens/held_epoch, span_tokens is rc2.1l's, ROUTE-VERDICT without the sgl-project#49 witness, no "WEG2 AGENT-SPAN sgl-project#49 on" line. Added: the sgl-project#49 rest (SGLANG_WEG2_FRONT_SPAN_INFLIGHT, FS 021272a) credits only when the span is on too -- record_inflight writes a held entry and would bring sgl-project#49's held-epoch price back under span off (mutant: removing the gate turns test_inflight_credits_only_under_the_span red). Tests (test27b.sh, CPU): new test_unify_agent_span_profile.py 14 (rows, per-profile default on env/SpanLRU/request_text, explicit wins, one environ entry, nextflash drops the held epoch, FS gate both ways); P49's test_weg2_agent_span_switch_p49 (default pin now via no form); the FS tests set the span on (27B form). 25 front/span/profile files: 431 passed, 13 failed -- the 13 identical on 61d9241 (no_route_1290 x2, s3s4_1233, sched_fix3 x4, scheduling_slice_a x6). profile_docker --check: AGENT_SPAN registry-only in every profile (no .env sets it); the one DIFF is the new measurement arm 27b-int8-x16 (--x-ceiling-tokens 16384, deliberate).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
Conflict in weg2/front.py (request_text gained tools_first next to the new front_span_inflight_line): both kept. The sgl-project#49 armed line now also names SGLANG_WEG2_ENABLE_AGENT_SPAN, which the in-flight credit needs as well ("WEG2-FRONT sgl-project#49 SPAN-INFLIGHT armed=<0|1> agent_span=<0|1> role=front pid="). Tests (desk, test27b.sh): test_weg2_prefix_trace_in + store_told 1400/1416e/hold_tk/positive_tk, twin_defer, inline_system_in_place, front_span_inflight_fs, arena_reset_release_h81, node_value_trail_1469, producer_phase_census_wiring_1061, hicache_arena_claim_1427, handoff_1442, arena_queue_refs_27b, p_intake_h91a, weg2/test_weg2_front*.py: 213 passed, 2 failed (test_weg2_front_lan_1321 TestFrontHostArgv x2 -- fail identically on b3180fb itself, launcher.py weg2_weight_source).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153) Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only. - pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted the admission-decision arc the whole file drove (_pp_send/_recv_admission_ decision, _pp_drain_voided_proxy, ...) and made the proxy channel unconditional; #1015c (206661a) retired the downstream refusal (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream pass refused, 0 served). The file now pins both halves of that retirement (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name). - pp_upstream_void_before_formation_951 x3 (found in the same family): the three refusal cases rewritten to the #1015c form (pass reaches the formation, predicate False on every shape, no void -> streak 0, slot kept). - scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency + SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the P->D flip D legitimately admits into; it now models the D->P flip as flip() runs it (awake stays 'D' until the end). Mutant (post-acquire re-check removed) -> red. t3 is the front defect, next commit. - sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst verdict into a local tested in the loop (b1d: locals assigned from a call/attribute count as state; b1e-g: the slice gets its stock values _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time` (real clock). Round-5 mutant (`if req is not None: continue` above the gate) still red on all four. - no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49 epoch keyword of price_remainder (196f6a8), H84 x_start_tokens (b895928). - s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request around _handle_generate_request_impl; the seam is the impl's first statement, the wrapper is pinned to clock-read + delegation. - pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants the usable-match floor on tree_cache; harness tree_cache=None.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…gl-project#49 armed lines Every prefix miss of an agent-load boot gets a token receipt. With SGLANG_WEG2_PREFIX_TRACE=1 (min rest SGLANG_WEG2_PREFIX_TRACE_MIN_TOKENS, default 1024), read once per process (weg2/prefix_trace.py, armed line "#PT PREFIX-TRACE armed=<0|1> min_tokens=M group=G pid=P"): - sgl-project#1420 WALK-STOP: one line per (rid, stop depth) with rest >= min, no process cap, root stops included; appends "rid=<full> trace=1 stop_node=N stop_hash=H want_run=[..] have_run=[[..]]". A split stop now names the TRUE rest and want (the un-advanced key made depth+remaining exceed pt-1 by the split's matched part; also untraced). - sgl-project#1400/sgl-project#1416/#1416d/#1416e (and TK's sgl-project#1442 ADOPT / TK ABS-TOLD): full rid, 8/256 sampling lifted. - sgl-project#1442 HANDOFF-KEYS REG, sgl-project#1040 EXTENT STATE-ALIGN: one line per distinct (rid, matched, span) / (rid, extent). - sgl-project#1469 EVICT: uncapped (outside the 600), + parent, pages, hash_first, hash_last, key_len, trace=1. - sgl-project#1427 ARENA-DROP: uncapped, + "trace=1 claim=<stem> dropped=<lo64 hex,..>". - sgl-project#1439 ARENA-PRESENT: once per distinct (keys, leading, first stem) with rest >= min, + "trace=1 break_stem_full=<stem>" (joinable to dropped=). No traced line sits on the decode round path. Armed lines (once per process): "#MZ INLINE-SYSTEM-IN-PLACE armed= merge= group= pid=" (anthropic adapter, P and D HTTP process) and "WEG2-FRONT sgl-project#49 SPAN-INFLIGHT armed= role=front pid=" (front start). Tests (desk, CUDA_VISIBLE_DEVICES="", test27b.sh): Base: desk/27b-unified-0926 @ 2bddf04 (rebased from f4b6f21, clean). - new test/registered/unit/mem_cache/test_weg2_prefix_trace_in.py: 16 passed - rebased run, all files below plus test_weg2_store_told_{hold,positive}_tk: 324 passed - managers/test_weg2_store_told_{1400,paced_1416e}, test_weg2_p_twin_defer, test_prefetch_deferral_1068, test_weg2_p_burst_clock_h42c, test_weg2_p_multi_anchor_tails_h42, anthropic/test_inline_system_in_place, mem_cache/test_arena_{queue_refs_27b,inner_anchor_release_0924}, test_backup_ack_drain_tp_asymmetry_h74, test_hicache_drain_budget_27b, weg2/test_weg2_{p_intake_h91a,front_span_inflight_fs,arena_reset_release_h81, node_value_trail_1469}, test_producer_phase_census_wiring_1061: 232 passed - test_hicache_arena_claim_1427, test_weg2_handoff_1442, test_weg2_store_short_tail_xsn437, test_evict_frees_the_bound_allocator_941, test_weg2_p_trim_end_anchor_0924: 50 passed - test_unified_radix_cache_unittest.py, per class, against the base: identical FAILED sets (pre-existing on the CPU desk; ps128_sw4/sw128 OOM in the 3 GiB cgroup on both trees) Unify note (RM, pick onto 2a7f992): conflicts line by line (store_told: PF fallback forget kept + IN _rt(rid); front: IN line helper + the P49 request_text signature). The sgl-project#49 SPAN-INFLIGHT armed line also names agent_span (the inflight credit needs the sgl-project#49 span, profile field agent_span); its test pin follows.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…e), no 1.3*X band -- behind SGLANG_WEG2_FRONT_EXACT_TOKENS (default off) User decision 26.09. ~19:00Z (memory d2p-sofort-flippen-und-x-exakt-0926): X holds EXACTLY for the pending tokens; the 1.3*X SHORT band (release table row 15, an open question, never built) is dropped. Measured, boot dkr27brc10bar1agent09261821 (front.log, 114 requests): chars/3 over- counted whole prompts by median 11.5 % (3.35 real chars/token, up to +60 %); D-direct error median +11, |median| 125, range -4481..+4273; misroutes: 1 SHORT really > X (weg2-10-7 3939/4780), 2 LONG really <= X (30-97, 30-106), ~6 LONG only by the chars/3 over-count. 9/78 D-direct under-priced by 1.0-4.7k: D resumes only at the Mamba anchor (sgl-project#988 LOADBACK anchor_depth = P's cached of the predecessor) -- a D cache fact, logged per request (X-EXACT-ERR), not banded. On: weg2/front_tokens.py builds D's OWN rendering stack in the front (AnthropicServing + OpenAIServingChat + TemplateManager) from the group's /get_server_info server args (tokenizer_path, chat_template, chat_template_default_kwargs, parsers), so the count is D's count_tokens code; pending = tokens - min(D's measurement, token LCP) (TokenSpans, sgl-project#1324/sgl-project#49 semantics in tokens, epoch-bound held credit). One worker thread, incremental per <|im_start|> segment (special token -> exact split). Lines: X-EXACT READY / PRICE / FALLBACK (not ready, timeout, multimodal, path) / TOKENS (front vs group count) / ERR. Off: nothing constructed or imported; ROUTE-VERDICT and X-ROUTE bytes unchanged. Cost (desk, 27B INT8): 160k tokens cold 302 ms, next turn 55 ms, event-loop max lag 31 ms; load 7 s, RSS +608 MiB first load. NF INT4 160k: cold 251, warm 57 ms. Tests: test_weg2_front_x_exact.py 21/21 (real tokenizers 27B INT8 + NF INT4-mixed: count == D count_tokens on real-text Claude-Code payloads, segmented == whole on 300 hostile strings, MZ inline-system parity, boundary X/X+1, credit, fallback, residual error log, switch off byte-identical). Affected suites (p49, sgl-project#49, inflight, messages pricing/usage, idle policy, dp wait, flipfast, vision economics, H75/H78, no_route_1290, x-split): 283 passed, 2 failed = test_weg2_no_route_1290 TheRouterEndToEnd x2, identical red on base c51591f (Front has no attribute 'vision'). dup_defs_gate exit 0.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…e), no 1.3*X band -- behind SGLANG_WEG2_FRONT_EXACT_TOKENS (default off) User decision 26.09. ~19:00Z (memory d2p-sofort-flippen-und-x-exakt-0926): X holds EXACTLY for the pending tokens; the 1.3*X SHORT band (release table row 15, an open question, never built) is dropped. Measured, boot dkr27brc10bar1agent09261821 (front.log, 114 requests): chars/3 over- counted whole prompts by median 11.5 % (3.35 real chars/token, up to +60 %); D-direct error median +11, |median| 125, range -4481..+4273; misroutes: 1 SHORT really > X (weg2-10-7 3939/4780), 2 LONG really <= X (30-97, 30-106), ~6 LONG only by the chars/3 over-count. 9/78 D-direct under-priced by 1.0-4.7k: D resumes only at the Mamba anchor (sgl-project#988 LOADBACK anchor_depth = P's cached of the predecessor) -- a D cache fact, logged per request (X-EXACT-ERR), not banded. On: weg2/front_tokens.py builds D's OWN rendering stack in the front (AnthropicServing + OpenAIServingChat + TemplateManager) from the group's /get_server_info server args (tokenizer_path, chat_template, chat_template_default_kwargs, parsers), so the count is D's count_tokens code; pending = tokens - min(D's measurement, token LCP) (TokenSpans, sgl-project#1324/sgl-project#49 semantics in tokens, epoch-bound held credit). One worker thread, incremental per <|im_start|> segment (special token -> exact split). Lines: X-EXACT READY / PRICE / FALLBACK (not ready, timeout, multimodal, path) / TOKENS (front vs group count) / ERR. Off: nothing constructed or imported; ROUTE-VERDICT and X-ROUTE bytes unchanged. Cost (desk, 27B INT8): 160k tokens cold 302 ms, next turn 55 ms, event-loop max lag 31 ms; load 7 s, RSS +608 MiB first load. NF INT4 160k: cold 251, warm 57 ms. Tests: test_weg2_front_x_exact.py 21/21 (real tokenizers 27B INT8 + NF INT4-mixed: count == D count_tokens on real-text Claude-Code payloads, segmented == whole on 300 hostile strings, MZ inline-system parity, boundary X/X+1, credit, fallback, residual error log, switch off byte-identical). Affected suites (p49, sgl-project#49, inflight, messages pricing/usage, idle policy, dp wait, flipfast, vision economics, H75/H78, no_route_1290, x-split): 283 passed, 2 failed = test_weg2_no_route_1290 TheRouterEndToEnd x2, identical red on base c51591f (Front has no attribute 'vision'). dup_defs_gate exit 0.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…xtflash aus) -- PF als Feld, aus Metall-Beleg dkr27brc10bar1agent09261821 (Image rc11a, 26.09. 18:21-18:56Z): 108 Requests, Flips 0,23 je Request (vorher 1,35), verschwendeter Prefill ~7 % (vorher 48 %), 0 Gruppentod, ENV-IM-RANG alle fuenf = 1 auf P, D und Front; #1416d TOLD-PROBE=8, PACED 8/8, TWIN 7/7, span_inflight 109. weg2/form.py: PREFIX_SWITCHES (Feld -> Env) und sechs ModelProfile-Felder inline_system_in_place SGLANG_ANTHROPIC_INLINE_SYSTEM_IN_PLACE (MZ, P==D Pflicht) told_probe_tree_key SGLANG_WEG2_TOLD_PROBE_TREE_KEY (TK; TOLD_ABSOLUTE folgt als Rang-Default) told_paced SGLANG_WEG2_TOLD_PACED (PX2) p_twin_defer SGLANG_WEG2_P_TWIN_DEFER (TW) front_span_inflight SGLANG_WEG2_FRONT_SPAN_INFLIGHT (sgl-project#49, wirkt nur mit agent_span) told_group_fallback SGLANG_WEG2_TOLD_GROUP_FALLBACK (PF; auf beiden Zeilen AUS, am Metall ungeprueft) qwen27b: die fuenf an, PF aus. nextflash: alle aus, bis NF freigibt. switch_defaults() fuehrt sie, damit profile_docker --check Drift sieht. Muster agent_span/standard_form: prefix_switch_state (explizite Env gewinnt, jede Schreibweise der name_compat-Familie; sonst Registry-Zeile; ohne Form aus), publish_prefix_switches (der eine Schreiber: build_env fuer P und D vor --env-p/--env-d, dieselbe Aufruf auf die Front-Env; ein Aus-Wert schreibt NICHTS -> NF byte-gleich), launcher-Zeile WEG2-PREFIX-SWITCHES mit Wert und Quelle (env | profile), Verweigerung bei MZ-Split ueber --env-p/--env-d. Rang-Seite: environ.py- Eintraege mit _profile_default (Invariante S2/S3), die direkten Leser (weg2_store_told, p_twin_defer, weg2_told_fallback) nehmen ihren Default ueber form.prefix_switch_armed, das explizite Parsing bleibt. Tests (test27b.sh, CUDA_VISIBLE_DEVICES=""): test_unify_prefix_switches_profile.py + agent_span/standard_form/S2/S3/S7/S8/S9 profile, build_env_group, name_compat_env_1b, form_axes, front_span_inflight_fs, told paced/pf/tk positive/hold, inline_system_in_place, prefix_trace_in: 313 passed. p_twin_defer, store_told_1400, prefetch_deferral_1068, p_burst_clock_h42c, p_multi_anchor_tails_h42, p_intake_h91a, launcher_argv_1235, vision_d_launcher, fp8_uniform_marlin, p_prefill_transient_114, rankdump_1378, hostsample_csv_1378: 200 passed. Parse-Ebene (FL4-Form, main-Vorlauf + build_env P/D + Front-Env, Basis 84183ca): nf.env Basis gegen neu: byte-gleich bis auf die Boot-Nonce (194 Zeilen). 27b-release-draft ohne _form-Zeilen (neu) == alte Fassung mit _form-Zeilen auf der Basis (172 Zeilen). 27b-Gruppen-Env ohne _form-Zeilen: +5 Schalter je P, D, Front gegen die Basis. profile_docker --check /spinning/gpu-arb/docker/profiles: TOTAL DIFF 1 (27b-int8-x16 --x-ceiling-tokens 16384, vorbestehend, auf der Basis ebenso); release-draft/27b/nf 0 DIFF.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153) Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only. - pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted the admission-decision arc the whole file drove (_pp_send/_recv_admission_ decision, _pp_drain_voided_proxy, ...) and made the proxy channel unconditional; #1015c (206661a) retired the downstream refusal (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream pass refused, 0 served). The file now pins both halves of that retirement (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name). - pp_upstream_void_before_formation_951 x3 (found in the same family): the three refusal cases rewritten to the #1015c form (pass reaches the formation, predicate False on every shape, no void -> streak 0, slot kept). - scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency + SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the P->D flip D legitimately admits into; it now models the D->P flip as flip() runs it (awake stays 'D' until the end). Mutant (post-acquire re-check removed) -> red. t3 is the front defect, next commit. - sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst verdict into a local tested in the loop (b1d: locals assigned from a call/attribute count as state; b1e-g: the slice gets its stock values _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time` (real clock). Round-5 mutant (`if req is not None: continue` above the gate) still red on all four. - no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49 epoch keyword of price_remainder (196f6a8), H84 x_start_tokens (b895928). - s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request around _handle_generate_request_impl; the seam is the impl's first statement, the wrapper is pinned to clock-read + delegation. - pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants the usable-match floor on tree_cache; harness tree_cache=None.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
… until measured), exact W31 re-queue, queue re-priced exactly for PK's needs_p Follow-up to c0347b7 (X holds exactly for the pending tokens, user 26.09.; no 1.3*X band). 1. Registry: ModelProfile.front_exact_tokens -> SGLANG_WEG2_FRONT_EXACT_TOKENS via PROFILE_SWITCH_DEFAULTS / _profile_default (both rows False until an agent-load boot measured X-EXACT-ERR / X-EXACT-TOKENS; explicit env wins; off without a form). 2. The W31 re-queue site (third Pending construction) priced the whole prompt by len(text)//3; under the switch it takes the front tokenizer's exact count. 3. The arrival price goes stale (a sgl-project#49 held credit lives only in its epoch, an in-flight twin credit or a D serve lands after the arrival, a measured zero retracts): Front._x_exact_reprice_queue re-prices every queued request whose price can still move -- at the epoch change, after record_inflight and after record_presence -- against the current TokenSpans. PK's phase_policy.needs_p / immediate_park_trigger read Pending.est_uncached, so the park fires on the CURRENT exact pending. Untouched: leg1_done, skip_leg1, p_only, x_requeues (D's own extent), chars/3-fallback prices. Only the number moves, routing flags do not. Line: WEG2 X-EXACT-REPRICE ... crossed=. Not built (operator): the Mamba-anchor residual (D resumes at an anchor below the credited prefix after an after_p serve) stays a logged X-EXACT-ERR, no front heuristic. Tests: test_weg2_front_x_exact.py 35/35 (+14: registry rows off, profile row/explicit matrix, twin credit reprices X+500 -> 200 and needs_p stops firing, epoch change raises the price over X and needs_p fires, measured/final prices untouched, off never reprices, W31 site exact). Affected suites (unify profile/switch/standard-form/anchor/prefix, 27B park immediate, p49, sgl-project#49, inflight, messages pricing/usage, idle policy, dp wait, flipfast, vision economics, H75/H78, no_route_1290, x-split): 459 passed, 0 failed. dup_defs_gate exit 0 (no new finding vs a4158ae).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 27, 2026
…stay off) -- P's store read of an NF session is no longer clamped to told=0 by a unigram anchor probe NF rc12t (dkrnfh91dprbar1dauer09271756, e1eb275, profile nf-h91-dpr): 0 of 41 P legs cached. P's PP0 read the store (weg2-1-9 18:05:34 "HiCache prefetch success ... loaded=54208", WEG2-LOAD-DEVICE 54336 tokens), then "sgl-project#1416 STORE-TOLD ANCHOR-CLAMP rid=weg2-1-9 completed=54336 anchored=0" -- told 0, the followers registered nothing (their STORE WITNESS unprobed), every rank prefilled from 0 (SERVED group=P prompt_tokens=54388 cached_tokens=0). Every P read with loaded>0 in that boot clamped to anchored=0. Root: with TOLD_PROBE_TREE_KEY off, weg2_store_told._anchor_clamp hashes origin_input_ids as UNIGRAM keys while the fetch keys the span as the tree does (RadixKey is_bigram=tree.is_eagle, bigram on NF's MTP group) and through the sgl-project#1442 hand-off chain -- the probe asked for keys nobody writes: the 27B #1416d class, fixed by TK, which the nextflash row kept off. The row now carries told_probe_tree_key, told_paced and p_twin_defer = True (TK brings TOLD_ABSOLUTE along). TK never without PACED: sgl-project#211 (rc11b, -15 % PP0 prefill) was TK's single-phase follower busy-wait in admission(). With PACED the only remaining wait is admission()'s bounded residual loop when a follower's read outlasts PP0's pacing window (max(1.25 x PP0's read, told x 1 s/100k), cap 10 s). On the NF P cut PP0 carries the heaviest store share (sgl-project#706 lines: KV slots 7/12 on PP0 vs 3 and 2; GDN blob 35.95 of 58.83 MB vs 13.07 and 9.81), so a follower's read ends inside the window; 27B metal 09271525: PACED-ADMIT 24, WAITED 0. PF (group fallback) stays off, MZ and front span inflight stay off (not released by the NF seat). An explicitly set env still wins; profiles/nf-h91-dpr-tk.env states the same values. Metal (NF seat, rc12u with nf-h91-dpr-tk, 19:09-19:16Z): P legs 10, 8 with a hit (the 2 cold ones first turns), walls 2.6-8.3 s instead of ~25 s; STORE-TOLD WAITED 0, waited_max 0.62 s, ANCHOR-CLAMP 0. Tests: test_unify_prefix_switches_profile.py updated to the new row (NF3 = TK, PACED, TW: registry, state per profile, published form, publish writes exactly the trio, build_env differs by exactly the trio, profile_docker facts, rank readers incl. TOLD_ABSOLUTE). Fix vs base dbeb9ff, 26 files one by one through test27b.sh (the prefix-switch/profile suites and every weg2_store_told / p_twin_defer / told_fallback suite): no new failure.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…D's weg2_resumable_depth (deepest held Mamba anchor) rc12z weg2-12-61: weg2-8-53 served ct=55424 (prompt 57396), RETAIN value=False at its finish; weg2-12-61 (57792 tokens) priced credit 55424 -> pending 2368 SHORT, D's extent 37952 -> X refusal midstream, 72 s P detour. rc12y weg2-16-42: weg2-13-41 ct=46848, no recurrent state after RETAIN value=False, weg2-16-42 priced 125. Front half of the NF-agreed seam: D sends weg2_resumable_depth (meta_info on /generate, sglext on the OpenAI wire, like weg2_prefill_s) at the leg-2 finish; the front records it beside each presence entry (SpanLRU and TokenSpans depth_caps, entry tuples unchanged) and credits min(credit, depth) -- the measured cached_tokens and the sgl-project#49 held prompt alike. 0 retracts. Field absent = the old credit, byte for byte. One line WEG2 PRESENCE-DEPTH-CAP rid=... credit old -> new when it caps; counters presence_depth_capped/uncapped/absent. Both leg-2 finish sites (stream: the finish chunk in the retained tail; body: js). Tests test_weg2_price_anchor_59_0928.py 12 passed: 12-61 (55424 -> 37952, pending 19840 > X 5074) and 16-42 (depth 0 -> whole 46851), both also WITHOUT the field reproducing the metal log lines (2368 / 125); held-epoch cap, inflight keeps the cap, cap trimmed with its entry, SpanLRU, both wires + stream tail, the log line, the wiring. Red-first on 7055ec7: 10 failed / 2 passed (the two no-field controls).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht. Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung). Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 28, 2026
…dit -- Bruecke f833fcb: D schrieb den Anker 37376 in 16-46s eigenem Extend ('sgl-project#1469 RETAIN rid=weg2-16-46 ... cache_len=37376', 19:50:38), 16-48 setzte darauf auf (19:50:51, '#cached-token: 37376'). Der D-Schlaf 19:51:15-16 verweigerte die Host-Sicherung ('sgl-project#1421 BACKUP-REFUSED why=mamba_pin ... pins=8/8', danach 'why=mamba_claim' hinter 'sgl-project#1427 ARENA-CLAIM REFUSED ... no free slot' der 32-Slot-Arena) und setzte den Baum zurueck ('sgl-project#1470 FLUSH-PUBLISH issued=13 unbacked_left=1', 'Cache flushed'). Nach dem Wake gab der Store den KV zurueck (store_hit=37376), aber keinen Anker ('[sgl-project#904 match-census] ... MambaComponent:absent=37376'); die Front preiste weg2-16-54 trotzdem SHORT (presence 35942 src=d_leg2_cached) -> 17 s D-Sitz, dann W31 am X-Gate (38291 Tok) und W50-Umweg ueber P (12 solche in rc12z26, 8 auf der Bruecke). Das X-Gate auf D preist schon die anker-nutzbare Tiefe (W31), die Front nicht. Fix: (a) Instrument: sgl-project#1421 BACKUP-REFUSED nennt depth= und rid= (weg2_node_depth, weg2_anchor_rid). (b) Der Flush zaehlt vor tree_cache.reset() jeden nur-Device-Mamba-Anker mit Tiefe (UnifiedRadixCache.weg2_unbacked_anchors, 'WEG2-ANCHOR-LOST at=flush n= depths= rids='); der Scheduler haelt die Tiefen bis zur Schlafantwort (weg2_take_anchors_lost, einmal lesen). (c) Die Gruppenzaun-Stimme des release-Legs traegt anchors_lost (Modulfunktion _weg2_anchors_lost_for_leg, [] auf jedem anderen Leg und ohne Scheduler-Ledger), die Reduktion vereinigt, ReleaseMemoryOccupationReqOutput.anchors_lost, der Tokenizer-Merge vereinigt (Feld fehlt, wenn nichts verloren -> alte Antwortform). (d) Front.flip liest nach completed_tags die Schlafantwort von D und nimmt jeden SpanLRU-Eintrag zurueck, dessen Kredit auf einer verlorenen Tiefe stand (depth_cap, sonst gemessene ct), 'WEG2 PRESENCE-ANCHOR-LOST'; ein Eintrag auf einem ueberlebenden Anker behaelt ihn. Zu viel Zuruecknehmen ist sicher (voller Preis = Preis des X-Gates). Die Rettung des Ankers ueber den Schlaf bleibt bei sgl-project#259 4c. Luecke: Arena-Verdraengung ohne L3 ausserhalb des Flushs erfasst der Ledger nicht (nur die Host-Kopie-Pruefung). Tests test_weg2_anchor_lost_presence_0928 (10): alle rot auf 1ea8133, gruen mit Fix. Nachbarn 18 Dateien (1424g, H113, H74, R12, sgl-project#59, X-exact, Spans, sgl-project#1470, sgl-project#49/P49, flip_cost_1235, 1273 x2, sched_fix3, W35, Profil, H91e, sgl-project#248): 298 gruen auf Fix und Basis, dieselben 5 Altlasten (VramCredit x3, sched_fix3 b4a/b4c).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…today's token spans, P-anchor presence off/on The witness exists (PREFILL-EINBRUCH-0929 K1, a59c95a, SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, default off) and never ran on 27B metal: 0 'WEG2 P-ANCHOR-PRESENCE' lines in every 27B boot. At the first content of an after_p leg 2 the front records P's END anchor (page floor of P's prompt) as a presence for every later text on that prefix. Fixture: fixtures/front_span_49/w109290020_events.json -- 993 events of the 30-min agent load (29.09. 00:20-00:54Z, bb82fbc): flips (epoch/awake), X-EXACT prices (tokens, reused prefix), route verdicts (X), P leg-1 and D leg-2 serves (cached/prompt/epoch/sgl-project#59 depth), leg-2 first content (via). The replay drives TokenSpans exactly as the front does (pending with the router's epoch rule, record_presence at the D finish, record_inflight at a d_direct first content, record_store_anchor at an after_p first content only with the switch on); token ids are synthesised from the reused prefixes. Result on today's code (the log predates K2, so it said 94 LONG of 182): OFF 58 LONG, ON 43 LONG -- 15 agent turns D-direct instead of P + 2 flips, none moved back. Four moved turns had a P side that held less than the credit (P leg-1 rest > X: weg2-20-21, 36-39, 42-42, 118-130); each one's source was RUNNING on D at the price time (first content given, leg 2 not finished), so D's radix held it. The test fails any other such move (danger direction W50). Metal (next 27B flip boot on the RC image with the flip-levers patch): 'WEG2 P-ANCHOR-PRESENCE' > 0, W50-REROUTE not above the off arm, tools/front_route_gap K1_FIXES -> ~0. Tests: test_weg2_p_anchor_presence_replay_w1_0930.py 4 green; test_weg2_price_store_anchor_0929 green.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…s its sgl-project#59 depth after D's sleep leg published it Switch SGLANG_WEG2_ENABLE_D_EPOCH_PUBLISH_PRESENCE (front only, default off). At a D->P flip done D's sleep leg has drained its HiCache writes, published the rest (sgl-project#1470) and joined the store queue (#1470b) before the kv pause. TokenSpans.promote_published(done_epoch): every text D served (held credit) in an epoch before done_epoch that carries D's own sgl-project#59 weg2_resumable_depth gets that depth as a STORE presence -- never the prompt, never past the text; no sgl-project#59 depth, no credit; the current epoch untouched. The depth is kept in TokenSpans.published BESIDE the entry: pending() credits it only where it lies on the new text's shared path, so the entry's lower measured anchor still credits a text that diverges earlier (folding it into ct lost weg2-134-146 in the replay). A new D reading of the text (a W31 refusal's small cached_tokens) pops it: the over-credit is refused, not repeated. Line 'WEG2 D-EPOCH-PUBLISH-PRESENCE', counters d_epoch_publish_presence[_tokens], then the queue reprice. Replay (fixture re-extracted: flip begin/done, D-side 'sgl-project#59 RESUMABLE' depths from the D log; the router's held-credit rule 'D awake AND serving'; a re-priced queued LONG reaches D only through the short drain, i.e. while D is awake and serving) on w109290020: OFF 59 LONG, L1 (P-anchor presence) 38 -- 21 moved (7 by the reprice at the witness), none back (supersedes the 58 -> 43 of 75833a7, which had no flip state, no D depths, no reprice); L2 alone 56 (3 moved: weg2-15-18, 27-27, 90-99), L1+L2 38 -- L2 adds NOTHING on top of L1 here: the 7 DDIRECT_GAP turns were priced DURING the D->P flip already under way (no held credit, no witness yet); their LONG route costs no extra flip, the flip happened anyway. Recommendation: L2 stays OFF (not in the flip-levers patch); L1 is the lever. Tests: test_weg2_d_epoch_publish_49_0930.py 5 (witnessed depth never the prompt; no depth / current epoch -> nothing; divergence keeps the lower anchor; DANGER: D's contradicting reading retracts -> LONG again; front hook only at D->P and switched on). Replay file 6 (+L2 arm). test_weg2_front_x_exact: the reprice-hook count 4 -> 5 with the new hook's guard pinned. Neighbours (40 files importing front_tokens / X-EXACT / flip): 0 new failures vs 75833a7.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…hared path (dual layout; P reports them in meta_info) Measured need (dual 0930 12:11, tools/front_route_gap): 11 of 18 LONG routes were repeated 40k/62k documents whose token LCP ends BEFORE the earlier text's end anchor; P hit 36863 / 61439 / 94207 -- its INNER anchors -- and the front credited 0 (presence_src=none, dual16 finding P2): the PX anchor rule knows the end anchor only, and the inner anchors sit at a DISTANCE of 4096 from the last one at chunk boundaries (weg2_anchor_step), not on a grid the front could compute. P side: mamba_component._weg2_anchor_step_declines records every INTERVAL anchor a request keeps (req._weg2_anchor_depths, raw token positions, rank-uniform inputs); the output streamer stamps them on the finishing output (req_time_stats.weg2_anchor_depths, in __getstate__ only when set); tokenizer_manager puts them into meta_info; serving_chat / serving_completions (SglExt.weg2_anchor_depths) and the Anthropic wire (AnthropicSglExt, _sglext_of) carry them to the front. Front: anchor_depths_of() reads them from P's leg-1 answer (Pending.leg1_anchor_depths); with the P-anchor witness (SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, D resumed P's END anchor at its first content) AND the new switch SGLANG_WEG2_ENABLE_INNER_ANCHOR_PRESENCE (default off) AND --dual-layout, the anchors below the end anchor -- only the deepest MAX_STATES_PER_PATH-1, the ones P's per-path cap keeps -- go into TokenSpans.inner beside the entry; pending() credits the deepest one <= the token LCP; a later D reading of the text pops them. Line 'WEG2 INNER-ANCHOR-PRESENCE', counter inner_anchor_presence. WHY DUAL ONLY: the 27B flip form releases inner anchors at P's reset (grid4096: SGLANG_WEG2_MAMBA_INNER_ANCHOR_RELEASE) -- crediting them there over-credits (W50). The premise that a D-direct turn reads P's published pages is K1's; the K1 metal verdict (W50-REROUTE) decides L3 too. Tests: test_weg2_inner_anchor_presence_49_0930.py 7 -- P records its interval anchors (1024-token chunks, anchors at 4096/8192/12288...), time_stats carry them, the Anthropic sglext carries them, the front reads them (sglext/meta_info), the deepest kept anchor on the shared path credits (36864 on a repeat diverging at 37500, was 0), one past the divergence credits nothing, a D reading retracts them, and the front records them only under the dual layout with the switch on. Neighbours (246 files naming the front / token spans / serving / tokenizer manager / mamba component / time stats): 0 new failures vs 3ec239f.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
… over P that the store would have let D serve One front log in, no GPU/tokenizer. Per LONG route: real_rest = P leg-1 prompt - P leg-1 cached; the source = the earlier request whose prompt equals this one's X-EXACT reused prefix; when it became a store presence (after_p first content = the P_ANCHOR_PRESENCE witness; a D-direct source only at its D leg-2 finish). Classes OK_LONG / K1_FIXES / INFLIGHT(after_p) / DDIRECT_GAP_served / OTHER. Real agent trace w109290020 (29.09. 00:20-00:54Z, bb82fbc, 30 min agent load): 185 routed turns, 96 over P (+2 flips each), 34 needed P (real rest > X=4096); of the 62 others: K1_FIXES 38, INFLIGHT 9, DDIRECT_GAP 7, OTHER 8. w109281851: 14 over P, 8 needed. Dual 0930 12:11 (P2 shape): 18 over P, 7 needed, 11 OTHER = repeated 40k/62k documents whose token LCP ends before the source's end anchor (P hit 36863 / 61439, front credit 0, presence_src=none). Test: test_weg2_front_route_gap_49_0930.py (2): one synthetic LONG per class, SHORT not counted.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…today's token spans, P-anchor presence off/on The witness exists (PREFILL-EINBRUCH-0929 K1, a59c95a, SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, default off) and never ran on 27B metal: 0 'WEG2 P-ANCHOR-PRESENCE' lines in every 27B boot. At the first content of an after_p leg 2 the front records P's END anchor (page floor of P's prompt) as a presence for every later text on that prefix. Fixture: fixtures/front_span_49/w109290020_events.json -- 993 events of the 30-min agent load (29.09. 00:20-00:54Z, bb82fbc): flips (epoch/awake), X-EXACT prices (tokens, reused prefix), route verdicts (X), P leg-1 and D leg-2 serves (cached/prompt/epoch/sgl-project#59 depth), leg-2 first content (via). The replay drives TokenSpans exactly as the front does (pending with the router's epoch rule, record_presence at the D finish, record_inflight at a d_direct first content, record_store_anchor at an after_p first content only with the switch on); token ids are synthesised from the reused prefixes. Result on today's code (the log predates K2, so it said 94 LONG of 182): OFF 58 LONG, ON 43 LONG -- 15 agent turns D-direct instead of P + 2 flips, none moved back. Four moved turns had a P side that held less than the credit (P leg-1 rest > X: weg2-20-21, 36-39, 42-42, 118-130); each one's source was RUNNING on D at the price time (first content given, leg 2 not finished), so D's radix held it. The test fails any other such move (danger direction W50). Metal (next 27B flip boot on the RC image with the flip-levers patch): 'WEG2 P-ANCHOR-PRESENCE' > 0, W50-REROUTE not above the off arm, tools/front_route_gap K1_FIXES -> ~0. Tests: test_weg2_p_anchor_presence_replay_w1_0930.py 4 green; test_weg2_price_store_anchor_0929 green.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…s its sgl-project#59 depth after D's sleep leg published it Switch SGLANG_WEG2_ENABLE_D_EPOCH_PUBLISH_PRESENCE (front only, default off). At a D->P flip done D's sleep leg has drained its HiCache writes, published the rest (sgl-project#1470) and joined the store queue (#1470b) before the kv pause. TokenSpans.promote_published(done_epoch): every text D served (held credit) in an epoch before done_epoch that carries D's own sgl-project#59 weg2_resumable_depth gets that depth as a STORE presence -- never the prompt, never past the text; no sgl-project#59 depth, no credit; the current epoch untouched. The depth is kept in TokenSpans.published BESIDE the entry: pending() credits it only where it lies on the new text's shared path, so the entry's lower measured anchor still credits a text that diverges earlier (folding it into ct lost weg2-134-146 in the replay). A new D reading of the text (a W31 refusal's small cached_tokens) pops it: the over-credit is refused, not repeated. Line 'WEG2 D-EPOCH-PUBLISH-PRESENCE', counters d_epoch_publish_presence[_tokens], then the queue reprice. Replay (fixture re-extracted: flip begin/done, D-side 'sgl-project#59 RESUMABLE' depths from the D log; the router's held-credit rule 'D awake AND serving'; a re-priced queued LONG reaches D only through the short drain, i.e. while D is awake and serving) on w109290020: OFF 59 LONG, L1 (P-anchor presence) 38 -- 21 moved (7 by the reprice at the witness), none back (supersedes the 58 -> 43 of 75833a7, which had no flip state, no D depths, no reprice); L2 alone 56 (3 moved: weg2-15-18, 27-27, 90-99), L1+L2 38 -- L2 adds NOTHING on top of L1 here: the 7 DDIRECT_GAP turns were priced DURING the D->P flip already under way (no held credit, no witness yet); their LONG route costs no extra flip, the flip happened anyway. Recommendation: L2 stays OFF (not in the flip-levers patch); L1 is the lever. Tests: test_weg2_d_epoch_publish_49_0930.py 5 (witnessed depth never the prompt; no depth / current epoch -> nothing; divergence keeps the lower anchor; DANGER: D's contradicting reading retracts -> LONG again; front hook only at D->P and switched on). Replay file 6 (+L2 arm). test_weg2_front_x_exact: the reprice-hook count 4 -> 5 with the new hook's guard pinned. Neighbours (40 files importing front_tokens / X-EXACT / flip): 0 new failures vs 75833a7. (cherry picked from commit 3ec239f, desk/27b-front-span-49-0930; environ.py: beside the RPC-STALL-WATCHDOG entry)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 30, 2026
…hared path (dual layout; P reports them in meta_info) Measured need (dual 0930 12:11, tools/front_route_gap): 11 of 18 LONG routes were repeated 40k/62k documents whose token LCP ends BEFORE the earlier text's end anchor; P hit 36863 / 61439 / 94207 -- its INNER anchors -- and the front credited 0 (presence_src=none, dual16 finding P2): the PX anchor rule knows the end anchor only, and the inner anchors sit at a DISTANCE of 4096 from the last one at chunk boundaries (weg2_anchor_step), not on a grid the front could compute. P side: mamba_component._weg2_anchor_step_declines records every INTERVAL anchor a request keeps (req._weg2_anchor_depths, raw token positions, rank-uniform inputs); the output streamer stamps them on the finishing output (req_time_stats.weg2_anchor_depths, in __getstate__ only when set); tokenizer_manager puts them into meta_info; serving_chat / serving_completions (SglExt.weg2_anchor_depths) and the Anthropic wire (AnthropicSglExt, _sglext_of) carry them to the front. Front: anchor_depths_of() reads them from P's leg-1 answer (Pending.leg1_anchor_depths); with the P-anchor witness (SGLANG_WEG2_ENABLE_P_ANCHOR_PRESENCE, D resumed P's END anchor at its first content) AND the new switch SGLANG_WEG2_ENABLE_INNER_ANCHOR_PRESENCE (default off) AND --dual-layout, the anchors below the end anchor -- only the deepest MAX_STATES_PER_PATH-1, the ones P's per-path cap keeps -- go into TokenSpans.inner beside the entry; pending() credits the deepest one <= the token LCP; a later D reading of the text pops them. Line 'WEG2 INNER-ANCHOR-PRESENCE', counter inner_anchor_presence. WHY DUAL ONLY: the 27B flip form releases inner anchors at P's reset (grid4096: SGLANG_WEG2_MAMBA_INNER_ANCHOR_RELEASE) -- crediting them there over-credits (W50). The premise that a D-direct turn reads P's published pages is K1's; the K1 metal verdict (W50-REROUTE) decides L3 too. Tests: test_weg2_inner_anchor_presence_49_0930.py 7 -- P records its interval anchors (1024-token chunks, anchors at 4096/8192/12288...), time_stats carry them, the Anthropic sglext carries them, the front reads them (sglext/meta_info), the deepest kept anchor on the shared path credits (36864 on a repeat diverging at 37500, was 0), one past the divergence credits nothing, a D reading retracts them, and the front records them only under the dual layout with the switch on. Neighbours (246 files naming the front / token spans / serving / tokenizer manager / mamba component / time stats): 0 new failures vs 3ec239f. (cherry picked from commit 93c2691, desk/27b-front-span-49-0930; front.leg1: beside the dual20 leg1_aborted line)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 2, 2026
…e D-Leg-2 kreditiert ihren End-Anker, auch ueber den D-Schlaf NF boot dkrnfint4bar1dauer10020634 (5b46b88), nextflash row agent_span=0: a FINISHED D leg 2 was credited only with its admission hit (record_presence dropped prompt_tokens/held_epoch). weg2-26-44 served 06:42:29,440 pt=53748 ct=50688 depth 53760; 44 ms later weg2-26-45 (SESSION-PREFIX common=53748) 'X-EXACT-PRICE pending=4582 credit=50688' > X=3647 -> LONG, P prefilled 1510 on a 53760 hit. weg2-24-40: credit 45824 (24-39's ct), 4339 > 3602, P 2291. (1) Same epoch: the nextflash row runs sgl-project#49 (form.py agent_span=True; released by the NF seat). Its premises hold on NF: the Flash-Next template renders <tools> first (chat_template.jinja 57-67), D sends weg2_resumable_depth on every finish (sgl-project#59 RESUMABLE mode=form-a-dcp-min). A served leg then holds min(prompt, D's depth) for its epoch (src=d_served_epoch): 26-45 credit 53696 pending 1574, 24-40 47808 / 2355 -- both SHORT. (2) Across epochs: the sgl-project#49 epoch guard protects the radix-only tokens past the anchor. With a sgl-project#59 depth the credit is min(prompt, depth, page floor) and survives D's sleep as a store load (src=d_served_anchor): in this boot 14 of 14 follow-ups after a D sleep read >= that anchor (26-45 P hit 53760 >= 53696, 27-46 55232, 64-104 on D 88336 >= 85888), 0 ANCHOR-LOST. The load is not priced: the LONG path loads the same prefix (P store read, D LOAD-DEVICE after the hand-off), X is a prefill bound. Without a depth the epoch guard stands. ANCHOR-LOST now also matches D's unclamped depth (TokenSpans.raw_depths; the K2 clamp had hidden it). (3) d_inflight (fceeda8) only for after_p legs (P's END-ANCHOR, realised): a prompt D prefilled itself anchors on its prefill track, 8 of 54 in-flight 'sgl-project#1469 RETAIN value=True' were below the page floor (weg2-4-7 19509 -> 17728); its finish reading covers it. Replay of the boot (25 LONG with real P prefill <= X): finished-same-epoch 9, cross-epoch 5, in-flight after_p 4 (attributed in that order; alone 9 / 8 / 11) = 18 SHORT. Tests: test_weg2_x_credit_finished_1002.py (11; 8 red on base: 26-45, 24-40, row, cross x3, no-depth guard, ANCHOR-LOST raw depth). Pins updated: agent_span profile rows (nextflash on), sgl-project#59 held-cap pin (outside the epoch 640 d_served_anchor), H61/H61b read the booked entry instead of SpanLRU.span_tokens (a chars/3 reader under sgl-project#49). Front-importing suites A+B (171 files) under the nextflash form: base 8+23 failed, fix 8+23 failed, 0 new reds (the 21 x_credit tests red on base only); no form: 507 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Oct 20, 2026
FLLIPER_MOE_MARLIN_C_SENTINEL=2 scans GEMM1 / ACT / GEMM2 rows for non-finite values. In the EP form (expert map, or a pool EM wave's filtered lanes) a -1 lane's GEMM1 / ACT rows are never written -- the GEMMs skip their blocks. _live_lanes_only folds a stage's bad (token, k) rows to tokens over LIVE lanes only (topk_ids >= 0) when is_ep; outside the EP form unchanged. With the sentinel on the arena is pre-filled with a finite value, so today no false alarm was reachable; this removes the dependency. The level-1 C-probe reads the output after GEMM2, whose cache is zeroed in the EP form: unaffected. Tests (serial, MemoryMax=2G, CUDA_VISIBLE_DEVICES=""): W2 + S51 + Marlin/sgl-project#49 suites 117 passed, 1 skipped; mutant (no exclusion) red.
TianHao65
added a commit
to TianHao65/sglang
that referenced
this pull request
Nov 3, 2026
Port the packaged workload drivers from PR sgl-project#13 onto sammysun0711/sglang:ci/qwen4-unified, record strict draft counters and weighted acceptance length, and retain the original GSM8K accuracy entry point. Add dataset validation, CPU regression checks, and a linked usage guide for issue sgl-project#49.
TianHao65
added a commit
to TianHao65/sglang
that referenced
this pull request
Nov 3, 2026
Port the DFlash acceptance changes developed on TianHao65/sglang:ci/qwen4-unified to the apinge/sglang qwen4_v0.5.21 target. Record strict draft counters and weighted acceptance length, preserve the original GSM8K accuracy entry point, and add dataset validation, CPU checks, and documentation for issue sgl-project#49.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1/completionsprotocol compatible with OpenAI API.v1/completions.\n\nso that it could be compatible with other frameworks.On my local, all tests in
tests/srt/test_httpserver_*andtests/srt/test_openai_server.pyare passed.