Repository navigation
fix: force max new tokens to be 1 for embedding request - #1019
Merged
Merged
Conversation
Ying1123
force-pushed
the
force-max-token
branch
from
August 10, 2024 20:23
43c5fbb to
5759825
Compare
Ying1123
force-pushed
the
force-max-token
branch
from
August 10, 2024 20:23
5759825 to
b954896
Compare
Ying1123
enabled auto-merge (squash)
August 10, 2024 20:46
Ying1123
disabled auto-merge
August 10, 2024 20:46
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
shun8686
pushed a commit
to shun8686/sglang
that referenced
this pull request
Jul 25, 2026
add test_npu_prefill_policy_bucket_balance.py
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…makespan, record why My sgl-project#1018 default was wrong and metal caught it. Three cuts, PP world tokens = min_r(available_r / (n_attn_r * 2048)): hand-pin [32,18,14] 616670 makespan [32,17,15] 592677 kv-floor [31,18,15] 567412 The solver had predicted kv-floor ABOVE makespan. The ranking INVERTED, and both solved cuts lost to the hand pin. Default is now makespan; kv-floor stays selectable for experiments and WARNS when selected. THE "IT SUMS INSTEAD OF TAKING THE MIN" HYPOTHESIS IS FALSE. world_kv_floor reduces with min(floor, cap). The defect is the QUANTITY reduced, not the reduction, and I checked before repeating the guess: * stage_kv_capacity charges the DUAL-PHASE arena, max(PP attention share, TP token share), because one arena backs both flip layouts. On this rig that bills stage1 for 4.75 attention-equivalents where the PP boot bills 4. * its spendable pot is the CUT GATE's -- what survives corridor, transient, fixed overhead and seam. Right for "does this cut fit", wrong for "how many tokens will the boot address": 2.33x below the boot's available bytes on the same cut and stage (my 265213 against metal 616670 on stage0). * it therefore cannot separate cuts sharing a binding stage: it returned the IDENTICAL 265213.33 for [32,18,14] and [32,17,15], which metal separates by 23993 tokens. PREDICTED BEFORE IT WAS BUILT, and the reference now lives in the docstrings so the next reader cannot miss it as I did: docs/dev/ANALYSE_799_token_axis_x_pp.md section 5.4 measured that on the CURRENT budgets PP0 is the POOR stage, so moving attention off it makes the min worse -- the historic "+33 % for [6,5,5]" inverts to -16 %. kv-floor moved an attention layer onto PP1 ([8,4,4] -> [7,5,4]) and PP1 fell 725191 -> 567412. That section also caps the token axis at 639800 (+6.51 %) at this cut. world_kv_floor is not deleted: it remains correct for the feasibility-scoped comparison inside the cut gate, which is what it was built for. Its docstring now says, in its own words, do not rank serving cuts with it. MEASURED SIDE-FINDING, and it is larger than the objective question. The kv-floor boot reports PP1 3.44 GB and PP2 2.12 GB available_gpu_mem -- 3280.6 and 2021.8 MiB, far ABOVE the 819-1229 MiB corridor band, which that law scores as a failed acceptance in its own right. Converting that unbuilt free at the poor ranks, which are exactly the ranks that set the world min: target free PP1 gain PP2 gain world min vs hand pin 1229 MiB +210088 +101477 674737 +58067 (+9.42 %) 1024 MiB +231080 +127717 674737 +58067 (+9.42 %) 819 MiB +252072 +153957 674737 +9.42 % The result is INSENSITIVE to the band target because PP0 becomes the binding rank in all three cases at 674737 -- its own capacity is untouched. So the prize is +9.42 % over the hand pin, which alone exceeds ANALYSE_799's +6.51 % ceiling; that ceiling held budgets fixed, and this is a different lever. AND IT REVERSES THE LESSON ABOUT kv-floor's DIRECTION. Once the free is converted PP0 binds, and the right move is then to shift attention OFF PP0 -- which is what kv-floor tried to do. kv-floor was not wrong in principle; it optimized for a state this boot does not currently reach because the free is never built. That is worth more than re-ranking cuts against today's state. Unit caveat, stated rather than smoothed: 3.44/2.12 were logged as "GB" and read here as decimal 1e9. If they are GiB the excess is larger and every conclusion above strengthens. Also available_gpu_mem is the CUDA runtime's free, not NVML's, and the corridor law is written on NVML free -- the ~424/518 MiB carve-out sits between them. DESK-PROVEN: default resolves to makespan -> [32,17,15]; kv-floor still resolves to [31,18,15] and emits the falsification warning; the trade line still prints both cuts on both axes. ruff unchanged (359).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…s ~2x and breaks serving d7c3736 (sgl-project#1015) was merged, booted, and REVERTED (a204832). It is not a mis-scoped refactor -- it takes serving down. MEASURED, boot_855_cons1_0840f82601_0830_073758 vs the same tree without it: TP-phase "weights + runtime state" (GiB) before sgl-project#1015 with sgl-project#1015 PP0 4.252 26.581 PP1 2.719 14.788 PP2 3.662 15.435 sum 10.633 56.804 The checkpoint is 27.5 GiB. sgl-project#1015 swung the post from under-booking (10.63, the defect I reported in sgl-project#1014) straight past correct to ~2x OVER-booking: 56.8 GiB of weights charged for a 27.5 GiB model, i.e. roughly the WHOLE model on EVERY rank instead of that rank's shard. Consequence, same boot: TP-phase PP0 KV budget "rest" 24.195 GiB -> 1.867 GiB EFFECTIVE max_total_num_tokens 615,434 -> 185,344 (-70 %) per-rank capacities 597210/448290/449378 -> 61152/57920/68724 coherence probe (/generate, greedy, "capital of France") -> NO RESPONSE The recalibrator then reacted to the collapsed capacities and moved the token vector [30,17,17] -> [21,20,23], which is correct behaviour on wrong inputs and would have written a garbage vector into the next boot's hint. REVERT VERIFIED: a204832 restores TP world 615,434 and the probe answers "Paris". One file, 60 lines (model_executor/model_runner_kv_cache_mixin.py) -- so the runtime blast radius of d7c3736 is exactly the weight post, which isolates the attribution. WHAT I GOT WRONG AND SHOULD HAVE CAUGHT FIRST: I supplied the sgl-project#1014 diff that motivated this fix, including the expected TP shard figures (~13.75/6.9/6.9 GiB from the 32,16,16 vector). I flagged at the time that I had NOT verified whether the flip re-materialises weights or re-views the PP residency, and that if it re-views them the "correct" TP post may legitimately be smaller than shard arithmetic. That caveat turned out to be the whole question, and a fix built on the arithmetic alone overshot. A boot-time falsifier -- assert the summed TP weight post lies between the largest single rank shard and the whole checkpoint -- would have caught this at the desk. The rest of the planner chain (#1009a/sgl-project#1010/sgl-project#1018/sgl-project#1019) is NOT reverted: it came in via cc70a6d and the earlier merges and is unaffected. ALSO ON THIS BOOT, and it works: sgl-project#778 cold-post-spill (5379340) is live and its markers fire on the real seam -- "phase-release LEND pp_to_tp: 65536 rows (~1024 MiB)" on PP0, 131072 rows (~1024 MiB) on PP1 and PP2 (row counts differ because the PP cell differs: 16384 B on PP0, 8192 B on PP1/PP2), "phase-release REPAY tp_to_pp: loan -> 0 rows" on all three before the prefill leg resumes. REPAY reports "0 MiB returned" -- the dial had not yet grown into the credit at that seam, so the repayment had nothing to hand back. That is the convergence behaviour the author described, not a leak, but it means the LEND's token effect is NOT yet demonstrated on metal and must not be counted in any ladder until a boot under load shows the pool actually taking it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
… time imbalance halves, at no capacity price QUEUE ITEM 2 -- DECISION BOOT, ANSWERED YES. Boot: /spinning/evidence-665-f1/boot_855_makespan_0840f82601_0830_093650.log Form: gdncov checkpoint, rank_mib 31800,18800,19800, cut [32,17,15] attn [8,4,4], with the sgl-project#769 instrument knob set (--regime-controller observe + SGLANG_REGIME_OBSERVE=1 + --kv-pressure-consensus-interval 1). `--pp-stage-ratio` is the deterministic route here: it is a VALIDATED OVERRIDE and MUTUALLY EXCLUSIVE with `--pp-solve-cut` (server_args.py: 16611-16621), so the solver cannot be asked for the same cut in one boot. THE QUESTION (§2 of the handover): the [32,18,14] cut showed a 21 % compute imbalance between two IDENTICAL 3080s -- the cut is balanced for memory, not for time. Does the makespan cut address it? MEASURED, compute-honest instrument (`Prefill rank batch ... (compute X, wait Y)`), FULL CHUNKS ONLY (4096 tok; the mode, not the max), n=29 samples per rank, 118,784 tokens seen by every rank: rank compute wait pure-compute rate PP0 (5090) 12,035.0 ms 0.0 ms 9,870 tok/s PP1 (3080) 30,900.2 ms 0.0 ms 3,844 tok/s <- pacer PP2 (3080) 28,492.9 ms 0.0 ms 4,169 tok/s 3080 imbalance: 8.45 % (was 20.53 % on [32,18,14], §2: 15,334/12,722) THE COMPARABLE QUANTITY IS THE RATIO, NOT THE ms. The two boots ran different token totals (118,784 vs 62,254) and different chunk mixes, so the absolute compute figures are not an A/B. The imbalance is dimensionless, same instrument, same rank pair, same hardware -- that one compares, and it more than halves. CAPACITY PRICE: NONE MEASURABLE. World (min over ranks) 592,677 vs 591,726 on the [32,18,14] std boot = +0.16 %, far inside the mandatory +/-5.3 % boot-to-boot band. The predicted -3.9 % pool delta did not show up above noise, so the cut is free on this axis. WAIT IS ZERO AGAIN: 0.0 ms on 105/105 rank-batch samples, now on a SECOND cut. The "collective floor eats the gain" thesis stays falsified for prefill on this boot form, and it is no longer a single-cut result. COHERENCE: /generate greedy "The capital of France is" -> " Paris." Serving is UP on this form. e2e ladder, unique-token-leading prompts (cache hit impossible by construction), wall-confounded instrument, stated as such: 12,024 tok in 13.241 s; 49,948 tok in 14.584 s. EVIDENCE TIER: BOOT-PROVEN, n=29 full chunks per rank. NOT CLAIMED: that the halved imbalance shows up as e2e throughput -- the 44 % unattributed residue of §2 is untouched and is queue item 3.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…del, nicht Gratis-Gewinn + die Posten-Zeile ist jetzt ehrlich == POSTEN 4: TP-VEKTOR-PROBE, drei Arme, PP-Cut [32,17,15] konstant == METHODIK ZUERST, weil sie das Ergebnis begrenzt. A/A-Rauschboden aus ZWEI Boots mit IDENTISCHEN Flags (makespan_093650 vs restore_100255): Rang PP0 643.198 -> 612.497 (-4,77%), PP1 729.329 -> 755.569 (+3,60%), PP2 592.677 -> 533.333 (-10,01%); TP-Pools alle -3,3..-3,8%. BEFUND, der eine stehende Regel praezisiert: das mandatierte +/-5,3%-Band gilt fuer EINZELNE Pool-Zahlen. Die WELT ist ein `min` ueber Raenge, erbt die Varianz des schlechtesten Rangs und wurde hier mit -10,01% gemessen. Ein Welt-Vergleich unter ~10% ist auf dieser Form nicht aufloesbar. SELBSTKORREKTUR dazu: mein Makespan-Commit (acac781) meldete "Welt 592.677 vs 591.726 = +0,16%, weit innerhalb +/-5,3%" als Beleg fuer "kein Kapazitaetspreis". Das Verdikt bleibt, die PRAEZISION war falsch -- +0,16% lag nie eng, es lag in einem Rauschen von 10%. DECODE-INSTRUMENT: n=5 warme Wiederholungen je Arm, erster Lauf nach Boot als WARMUP verworfen (er war reproduzierbar kalt: Arm 2 rep1 47,4 gegen Median 56,9). Mein erster Arm-1-Boden von "1,71%" aus n=2 war ein Gluecks-Paar und als Boden wertlos -- Stichprobenfehler, hier korrigiert. Arm tp-vector Welt (min) Decode Median Spreizung n_warm 1 32,16,16 592.677 / 53,5 / 54,4 (n=2, unter- 533.333 (A/A) abgetastet) 2 34,15,15 559.395 56,9 tok/s 6,2% 5 3 42,11,11 374.755 60,1 tok/s 0,6% 5 VERDIKT: * Arm 1 vs Arm 2: NICHT TRENNBAR auf beiden Achsen. Arm 2s Welt 559.395 liegt INNERHALB des A/A-Intervalls von Arm 1 (533.333..592.677); die Decode-Mediane liegen ineinander. Wie vorhergesagt: kv_heads [2,1,1] ist ueber alle Kandidaten invariant, die KV-Zelle bewegt sich nicht. * Arm 3: TRENNBAR UND EIN HANDEL, kein Gewinn. Welt 374.755 = -33% gegen Arm 2, weit ausserhalb jedes Bandes, gegen +5,6% Decode-Median. Richtung plausibel (42/64 der Gewichtsmasse auf PP0 = der schnellsten Karte, also schnellerer Decode; und derselbe Grund frisst PP0s KV-Raum, und PP0 ist der weltbindende Rang). Fuer einen Strang, dessen Preis Kapazitaet ist, ist das ein schlechter Tausch. * EMPFEHLUNG: bei 32,16,16 bleiben. Kein Kandidat kauft Welt. SCHRANKEN: Arm-1-Decode ist unterabgetastet (n=2, anderes Protokoll), also KEINE praezise Arm1-vs-Arm3-Differenz behauptet. Welt je Arm ist ein Einzelboot; nur Arm 3s -33% ueberschreitet den gemessenen 10%-Boden. Die Decode-Spreizungen selbst differieren stark (6,2% vs 0,6%), die Varianz ist also zustandsabhaengig -- der 5,6%-Abstand liegt an der Aufloesungsgrenze. Der 34,15,15-q-head-Confound der Uebergabe ist damit gegenstandslos: der Arm ist ohnehin nicht separierbar. == sgl-project#1026 EINSCHUB: die Posten-Zeile misst ihr Total jetzt UNABHAENGIG == Nur Anzeige-Ehrlichkeit. Die Planner-Konsum-Haelfte (:1000-1013 CONSUMES) bleibt unangetastet -- sgl-project#1015-Terrain. Bauform uebernommen von den beiden Instrumenten, die es im Baum richtig machen: Total unabhaengig messen (torch.cuda.mem_get_info -> echtes CUDA-free, dieselbe Quelle, der kv_backing_relief._free_bytes vertraut, nie total-minus-used) PLUS expliziter Rest-Term (logindex flips-Form). ERSTE ZEILE AM METALL (boot_855_r1026_..._102721.log): PP0 rest=13.875 measured free=16.888 unaccounted=+3.013 PP1 rest= 8.440 measured free=12.057 unaccounted=+3.617 PP2 rest= 7.898 measured free=10.405 unaccounted=+2.507 Das ist GENAU die sgl-project#1021-Capstone-Groesse, die die Uebergabe als "unchanged and unused across all passes" fuehrt -- die Sizing-Zeile druckt sie ab jetzt selbst, je Rang, auf jedem Boot. Die Reihenfolge des Koordinators (sgl-project#1026 VOR sgl-project#1021) war damit richtig: sgl-project#1021 haette auf einer Zeile gerechnet, die diesen Term nicht kannte. SELBSTKORREKTUR im selben Zug: mein erster Kommentar deutete ein positives Residuum als "KV-Pool ohne Gegenwert verschenkt". FALSCH. Der Ueberschuss ist der UNGEBUCHTE Topf, aus dem die spaeteren Allokationen tatsaechlich bedient werden -- die Graph-Capture (810 MiB) und die Arena-Tails kommen von genau hier. Damit ist die Zahl die ehrliche Kopffreiheit und zugleich die ganze sgl-project#1025-Geschichte in einer Groesse: gdncov bootet, weil hier 2,5-3,6 GiB stehen (weit ueber den 810 MiB der Capture); das lm_head-Artefakt stirbt, weil seine freigewordenen Gewichtsbytes in KV-Pool verwandelt werden und GENAU DIESEN Ueberschuss unter den Capture-Bedarf druecken. Ein schrumpfendes Residuum ueber Boots ist die Fruehwarnung, die vier Boots der sgl-project#855-Leiter sich haben erarbeiten muessen. Kommentar entsprechend korrigiert. ZUSTAND: Serving oben auf 32,16,16 + Makespan-Cut, health 200, greedy "capital of France" -> " Paris.", Deadman armiert. BELEG-STUFE: BOOT-GEMESSEN. Kein Suiten-Lauf (Speed-Modus); Abnahme-Suite ist Posten 6.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…TIGT -- der Makespan-Cut bezahlt Balance mit Welt. Posten-2-Verdikt KORRIGIERT. Einschub-Boot boot_855_arena3218_0840f82601_0830_103720.log, [32,18,14]/[8,4,4], sonst byte-gleiche Flags, sgl-project#1026-Zeile und Regime-Instrument mitgefahren. Verdikt-Regeln (a)-(d) des Koordinators vorab festgenagelt, hier abgearbeitet. == (a) PRIMAERE LESEGROESSEN, [32,18,14] gegen [32,17,15] == Groesse [32,18,14] [32,17,15] Delta PP2 Arena-Tail 1126.95 MiB 1493.22 MiB -366 MiB (-24,5%) PP1 Arena-Tail 167.02 MiB 0.00 MiB PP0 Arena-Tail 436.45 MiB 436.45 MiB unveraendert PP2 Pool (der Binder) 637.726 533.333 -16,4% Welt (min ueber Raenge) 618.778 533.333 -13,8% Bindender Rang PP0 PP2 WECHSELT unaccounted wr0/1/2 +3.013/+3.684/+2.440 +3.013/+3.617/+2.507 == (b) VERDIKT: BESTAETIGT, beide Bedingungen erfuellt == PP2s Arena-Tail faellt materiell (-24,5%), UND die Welt ist trennbar: -13,8% liegt ueber dem gemessenen ~10%-Welt-Varianzboden, und am BINDER ist es eindeutig -- PP2s Pool ueber zwei Boots je Arm: [32,17,15]: 533.333 / 592.677 (Spanne 533-593k) [32,18,14]: 637.726 / 635.826 (Spanne 636-638k, 0,3% auseinander) Die Spannen ueberlappen NICHT, und die [32,18,14]-Stichprobe ist eng. Der Arena-Tail selbst ist ohnehin keine verrauschte Groesse (die Probe meldet derived == measured, Fehler 0.00). == (c) DER PREIS DER GEGENRICHTUNG, wie verlangt beziffert == Gleiche Ladder wie beim Makespan-Boot (74k/12k/50k), volle Chunks (4096): Rang [32,18,14] n=32 [32,17,15] n=29 PP0 13.271,8 ms 12.035,0 ms PP1 34.858,4 ms Pacer 30.900,2 ms Pacer PP2 29.869,7 ms 28.492,9 ms 3080-Imbalance 16,70% 8,45% Pacer-Rate 3.760 tok/s 3.844 tok/s (+2,2%) Der Makespan-Cut HALBIERT die Imbalance-Kennzahl -- das war richtig gemessen. Aber der Pacer gewinnt dabei nur 2,2%, und das liegt INNERHALB des Decode-/Mess-Rauschens dieses Rigs. Die Kennzahl bewegt sich, der Durchsatz nicht. Bezahlt werden dafuer 13,8% Welt. == SELBSTKORREKTUR (vierte heute, und die folgenreichste) == Mein Posten-2-Commit acac781 verkuendete den Makespan-Cut als Gewinn "at no capacity price", gestuetzt auf Welt 592.677 gegen 591.726 = +0,16%. Dieser Vergleich war WERTLOS: er lag in einem Rauschen von ~10%, das ich selbst erst danach gemessen habe (A/A-Paar), und er war zusaetzlich durch die Seam-Record-Vererbung konfundiert. Sauber gegengetestet kostet der Cut 13,8% Welt und 16,4% am Binder. Das Verdikt "Makespan-Cut ist ein Gewinn" ist HIERMIT ZURUECKGEZOGEN. Es war ein Handel, und auf einem Kapazitaets-Strang ist es derselbe schlechte Handel, den ich beim TP-Vektor-Arm 42,11,11 abgelehnt habe -- ich habe ihn dort erkannt und hier nicht, weil dort die Zahl gross war und hier im Rauschen lag. == EMPFEHLUNG FUER DIE SHIPPING-FORM: [32,18,14] == Serving steht bereits darauf (dieser Boot). Die Abnahme-Suite sollte auf der Form laufen, die wir ausliefern -- also auf dieser, nicht auf dem Makespan-Cut. == (d) MESSBASIS FUER sgl-project#1019 == Der Objective-Rebuild des Planner-Agenten bekommt hiermit einen zweiten Datenpunkt, an dem BEIDE Objectives falsch liegen: makespan waehlt [32,17,15] und optimiert eine Kennzahl, die sich nicht in Pacer-Durchsatz uebersetzt, waehrend die Welt 13,8% faellt. Die fehlende Groesse im Kostenmodell ist der ARENA-TAIL: er ist cut-abhaengig, deterministisch und trifft genau den Rang, der die Welt bindet. Ich baue daran NICHT weiter. BELEG-STUFE: BOOT-GEMESSEN, n=32 volle Chunks je Rang fuer Compute, n=2 Boots je Arm fuer den Binder-Pool, n=1 je Arm fuer Arena-Tail (deterministische Groesse). Kein Suiten-Lauf.
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
…ex (the same cuSOLVER potrf without the host info check) and an fp64 round-robin Jacobi eigh kernel (the fork's graph-safe decode solver widened to fp64, fixed sweeps, eigh's ascending order); whole-layer prefill commit graph back on for init_method=k31 (SGLANG_GDN_K31_EIGH=torch keeps the eager reference call); CPU tests vs torch.linalg.eigh (top-8 projector < 1e-12, all splits within the Davis-Kahan bound), GPU check (capture, bitwise replay, timing) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 27, 2026
…follow-up) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.