Skip to content

fix: force max new tokens to be 1 for embedding request - #1019

Merged
Ying1123 merged 1 commit into
mainfrom
force-max-token
Aug 10, 2024
Merged

Ying1123 merged 1 commit into
mainfrom
force-max-token

Conversation

@Ying1123

Copy link
Copy Markdown
Contributor

No description provided.

@Ying1123
Ying1123 enabled auto-merge (squash) August 10, 2024 20:46
@Ying1123
Ying1123 disabled auto-merge August 10, 2024 20:46
@Ying1123
Ying1123 merged commit b68c4c0 into main Aug 10, 2024
@Ying1123
Ying1123 deleted the force-max-token branch August 10, 2024 20:46
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
shun8686 pushed a commit to shun8686/sglang that referenced this pull request Jul 25, 2026
add test_npu_prefill_policy_bucket_balance.py
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…makespan, record why

My sgl-project#1018 default was wrong and metal caught it. Three cuts, PP world tokens
= min_r(available_r / (n_attn_r * 2048)):

  hand-pin [32,18,14]  616670
  makespan [32,17,15]  592677
  kv-floor [31,18,15]  567412

The solver had predicted kv-floor ABOVE makespan. The ranking INVERTED, and
both solved cuts lost to the hand pin. Default is now makespan; kv-floor stays
selectable for experiments and WARNS when selected.

THE "IT SUMS INSTEAD OF TAKING THE MIN" HYPOTHESIS IS FALSE. world_kv_floor
reduces with min(floor, cap). The defect is the QUANTITY reduced, not the
reduction, and I checked before repeating the guess:

* stage_kv_capacity charges the DUAL-PHASE arena, max(PP attention share, TP
  token share), because one arena backs both flip layouts. On this rig that
  bills stage1 for 4.75 attention-equivalents where the PP boot bills 4.
* its spendable pot is the CUT GATE's -- what survives corridor, transient,
  fixed overhead and seam. Right for "does this cut fit", wrong for "how many
  tokens will the boot address": 2.33x below the boot's available bytes on the
  same cut and stage (my 265213 against metal 616670 on stage0).
* it therefore cannot separate cuts sharing a binding stage: it returned the
  IDENTICAL 265213.33 for [32,18,14] and [32,17,15], which metal separates by
  23993 tokens.

PREDICTED BEFORE IT WAS BUILT, and the reference now lives in the docstrings
so the next reader cannot miss it as I did:
docs/dev/ANALYSE_799_token_axis_x_pp.md section 5.4 measured that on the
CURRENT budgets PP0 is the POOR stage, so moving attention off it makes the
min worse -- the historic "+33 % for [6,5,5]" inverts to -16 %. kv-floor moved
an attention layer onto PP1 ([8,4,4] -> [7,5,4]) and PP1 fell 725191 ->
567412. That section also caps the token axis at 639800 (+6.51 %) at this cut.

world_kv_floor is not deleted: it remains correct for the feasibility-scoped
comparison inside the cut gate, which is what it was built for. Its docstring
now says, in its own words, do not rank serving cuts with it.

MEASURED SIDE-FINDING, and it is larger than the objective question.
The kv-floor boot reports PP1 3.44 GB and PP2 2.12 GB available_gpu_mem --
3280.6 and 2021.8 MiB, far ABOVE the 819-1229 MiB corridor band, which that
law scores as a failed acceptance in its own right. Converting that unbuilt
free at the poor ranks, which are exactly the ranks that set the world min:

  target free   PP1 gain      PP2 gain      world min   vs hand pin
  1229 MiB      +210088       +101477       674737      +58067  (+9.42 %)
  1024 MiB      +231080       +127717       674737      +58067  (+9.42 %)
   819 MiB      +252072       +153957       674737      +9.42 %

The result is INSENSITIVE to the band target because PP0 becomes the binding
rank in all three cases at 674737 -- its own capacity is untouched. So the
prize is +9.42 % over the hand pin, which alone exceeds ANALYSE_799's +6.51 %
ceiling; that ceiling held budgets fixed, and this is a different lever.

AND IT REVERSES THE LESSON ABOUT kv-floor's DIRECTION. Once the free is
converted PP0 binds, and the right move is then to shift attention OFF PP0 --
which is what kv-floor tried to do. kv-floor was not wrong in principle; it
optimized for a state this boot does not currently reach because the free is
never built. That is worth more than re-ranking cuts against today's state.

Unit caveat, stated rather than smoothed: 3.44/2.12 were logged as "GB" and
read here as decimal 1e9. If they are GiB the excess is larger and every
conclusion above strengthens. Also available_gpu_mem is the CUDA runtime's
free, not NVML's, and the corridor law is written on NVML free -- the ~424/518
MiB carve-out sits between them.

DESK-PROVEN: default resolves to makespan -> [32,17,15]; kv-floor still
resolves to [31,18,15] and emits the falsification warning; the trade line
still prints both cuts on both axes. ruff unchanged (359).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…s ~2x and breaks serving

d7c3736 (sgl-project#1015) was merged, booted, and REVERTED (a204832). It is not a
mis-scoped refactor -- it takes serving down.

MEASURED, boot_855_cons1_0840f82601_0830_073758 vs the same tree without it:

  TP-phase "weights + runtime state" (GiB)   before sgl-project#1015   with sgl-project#1015
    PP0                                          4.252        26.581
    PP1                                          2.719        14.788
    PP2                                          3.662        15.435
    sum                                         10.633        56.804

  The checkpoint is 27.5 GiB. sgl-project#1015 swung the post from under-booking (10.63,
  the defect I reported in sgl-project#1014) straight past correct to ~2x OVER-booking:
  56.8 GiB of weights charged for a 27.5 GiB model, i.e. roughly the WHOLE
  model on EVERY rank instead of that rank's shard.

  Consequence, same boot:
    TP-phase PP0 KV budget "rest"    24.195 GiB  ->   1.867 GiB
    EFFECTIVE max_total_num_tokens    615,434    ->   185,344   (-70 %)
    per-rank capacities         597210/448290/449378 -> 61152/57920/68724
    coherence probe (/generate, greedy, "capital of France")  -> NO RESPONSE

  The recalibrator then reacted to the collapsed capacities and moved the
  token vector [30,17,17] -> [21,20,23], which is correct behaviour on wrong
  inputs and would have written a garbage vector into the next boot's hint.

REVERT VERIFIED: a204832 restores TP world 615,434 and the probe answers
"Paris". One file, 60 lines
(model_executor/model_runner_kv_cache_mixin.py) -- so the runtime blast radius
of d7c3736 is exactly the weight post, which isolates the attribution.

WHAT I GOT WRONG AND SHOULD HAVE CAUGHT FIRST: I supplied the sgl-project#1014 diff that
motivated this fix, including the expected TP shard figures (~13.75/6.9/6.9
GiB from the 32,16,16 vector). I flagged at the time that I had NOT verified
whether the flip re-materialises weights or re-views the PP residency, and
that if it re-views them the "correct" TP post may legitimately be smaller
than shard arithmetic. That caveat turned out to be the whole question, and a
fix built on the arithmetic alone overshot. A boot-time falsifier -- assert
the summed TP weight post lies between the largest single rank shard and the
whole checkpoint -- would have caught this at the desk.

The rest of the planner chain (#1009a/sgl-project#1010/sgl-project#1018/sgl-project#1019) is NOT reverted: it
came in via cc70a6d and the earlier merges and is unaffected.

ALSO ON THIS BOOT, and it works: sgl-project#778 cold-post-spill (5379340) is live and
its markers fire on the real seam --
  "phase-release LEND pp_to_tp: 65536 rows (~1024 MiB)" on PP0,
  131072 rows (~1024 MiB) on PP1 and PP2 (row counts differ because the PP
  cell differs: 16384 B on PP0, 8192 B on PP1/PP2),
  "phase-release REPAY tp_to_pp: loan -> 0 rows" on all three before the
  prefill leg resumes.
REPAY reports "0 MiB returned" -- the dial had not yet grown into the credit
at that seam, so the repayment had nothing to hand back. That is the
convergence behaviour the author described, not a leak, but it means the
LEND's token effect is NOT yet demonstrated on metal and must not be counted
in any ladder until a boot under load shows the pool actually taking it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
… time imbalance halves, at no capacity price

QUEUE ITEM 2 -- DECISION BOOT, ANSWERED YES.
Boot: /spinning/evidence-665-f1/boot_855_makespan_0840f82601_0830_093650.log
Form: gdncov checkpoint, rank_mib 31800,18800,19800, cut [32,17,15] attn
[8,4,4], with the sgl-project#769 instrument knob set (--regime-controller observe +
SGLANG_REGIME_OBSERVE=1 + --kv-pressure-consensus-interval 1).
`--pp-stage-ratio` is the deterministic route here: it is a VALIDATED
OVERRIDE and MUTUALLY EXCLUSIVE with `--pp-solve-cut` (server_args.py:
16611-16621), so the solver cannot be asked for the same cut in one boot.

THE QUESTION (§2 of the handover): the [32,18,14] cut showed a 21 % compute
imbalance between two IDENTICAL 3080s -- the cut is balanced for memory, not
for time. Does the makespan cut address it?

MEASURED, compute-honest instrument (`Prefill rank batch ... (compute X,
wait Y)`), FULL CHUNKS ONLY (4096 tok; the mode, not the max), n=29 samples
per rank, 118,784 tokens seen by every rank:

  rank         compute     wait    pure-compute rate
  PP0 (5090)   12,035.0 ms  0.0 ms   9,870 tok/s
  PP1 (3080)   30,900.2 ms  0.0 ms   3,844 tok/s  <- pacer
  PP2 (3080)   28,492.9 ms  0.0 ms   4,169 tok/s

  3080 imbalance: 8.45 %   (was 20.53 % on [32,18,14], §2: 15,334/12,722)

THE COMPARABLE QUANTITY IS THE RATIO, NOT THE ms. The two boots ran
different token totals (118,784 vs 62,254) and different chunk mixes, so
the absolute compute figures are not an A/B. The imbalance is dimensionless,
same instrument, same rank pair, same hardware -- that one compares, and it
more than halves.

CAPACITY PRICE: NONE MEASURABLE. World (min over ranks) 592,677 vs 591,726
on the [32,18,14] std boot = +0.16 %, far inside the mandatory +/-5.3 %
boot-to-boot band. The predicted -3.9 % pool delta did not show up above
noise, so the cut is free on this axis.

WAIT IS ZERO AGAIN: 0.0 ms on 105/105 rank-batch samples, now on a SECOND
cut. The "collective floor eats the gain" thesis stays falsified for prefill
on this boot form, and it is no longer a single-cut result.

COHERENCE: /generate greedy "The capital of France is" -> " Paris." Serving
is UP on this form.
e2e ladder, unique-token-leading prompts (cache hit impossible by
construction), wall-confounded instrument, stated as such: 12,024 tok in
13.241 s; 49,948 tok in 14.584 s.

EVIDENCE TIER: BOOT-PROVEN, n=29 full chunks per rank.
NOT CLAIMED: that the halved imbalance shows up as e2e throughput -- the
44 % unattributed residue of §2 is untouched and is queue item 3.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…del, nicht Gratis-Gewinn + die Posten-Zeile ist jetzt ehrlich

== POSTEN 4: TP-VEKTOR-PROBE, drei Arme, PP-Cut [32,17,15] konstant ==

METHODIK ZUERST, weil sie das Ergebnis begrenzt. A/A-Rauschboden aus ZWEI
Boots mit IDENTISCHEN Flags (makespan_093650 vs restore_100255):
  Rang PP0 643.198 -> 612.497 (-4,77%), PP1 729.329 -> 755.569 (+3,60%),
  PP2 592.677 -> 533.333 (-10,01%); TP-Pools alle -3,3..-3,8%.
BEFUND, der eine stehende Regel praezisiert: das mandatierte +/-5,3%-Band
gilt fuer EINZELNE Pool-Zahlen. Die WELT ist ein `min` ueber Raenge, erbt
die Varianz des schlechtesten Rangs und wurde hier mit -10,01% gemessen.
Ein Welt-Vergleich unter ~10% ist auf dieser Form nicht aufloesbar.
SELBSTKORREKTUR dazu: mein Makespan-Commit (acac781) meldete "Welt
592.677 vs 591.726 = +0,16%, weit innerhalb +/-5,3%" als Beleg fuer
"kein Kapazitaetspreis". Das Verdikt bleibt, die PRAEZISION war falsch --
+0,16% lag nie eng, es lag in einem Rauschen von 10%.

DECODE-INSTRUMENT: n=5 warme Wiederholungen je Arm, erster Lauf nach Boot
als WARMUP verworfen (er war reproduzierbar kalt: Arm 2 rep1 47,4 gegen
Median 56,9). Mein erster Arm-1-Boden von "1,71%" aus n=2 war ein
Gluecks-Paar und als Boden wertlos -- Stichprobenfehler, hier korrigiert.

  Arm  tp-vector   Welt (min)   Decode Median   Spreizung   n_warm
  1    32,16,16    592.677 /               53,5 / 54,4      (n=2, unter-
                   533.333 (A/A)                             abgetastet)
  2    34,15,15    559.395      56,9 tok/s      6,2%         5
  3    42,11,11    374.755      60,1 tok/s      0,6%         5

VERDIKT:
* Arm 1 vs Arm 2: NICHT TRENNBAR auf beiden Achsen. Arm 2s Welt 559.395
  liegt INNERHALB des A/A-Intervalls von Arm 1 (533.333..592.677); die
  Decode-Mediane liegen ineinander. Wie vorhergesagt: kv_heads [2,1,1] ist
  ueber alle Kandidaten invariant, die KV-Zelle bewegt sich nicht.
* Arm 3: TRENNBAR UND EIN HANDEL, kein Gewinn. Welt 374.755 = -33% gegen
  Arm 2, weit ausserhalb jedes Bandes, gegen +5,6% Decode-Median. Richtung
  plausibel (42/64 der Gewichtsmasse auf PP0 = der schnellsten Karte, also
  schnellerer Decode; und derselbe Grund frisst PP0s KV-Raum, und PP0 ist
  der weltbindende Rang). Fuer einen Strang, dessen Preis Kapazitaet ist,
  ist das ein schlechter Tausch.
* EMPFEHLUNG: bei 32,16,16 bleiben. Kein Kandidat kauft Welt.
SCHRANKEN: Arm-1-Decode ist unterabgetastet (n=2, anderes Protokoll), also
KEINE praezise Arm1-vs-Arm3-Differenz behauptet. Welt je Arm ist ein
Einzelboot; nur Arm 3s -33% ueberschreitet den gemessenen 10%-Boden. Die
Decode-Spreizungen selbst differieren stark (6,2% vs 0,6%), die Varianz ist
also zustandsabhaengig -- der 5,6%-Abstand liegt an der Aufloesungsgrenze.
Der 34,15,15-q-head-Confound der Uebergabe ist damit gegenstandslos: der Arm
ist ohnehin nicht separierbar.

== sgl-project#1026 EINSCHUB: die Posten-Zeile misst ihr Total jetzt UNABHAENGIG ==

Nur Anzeige-Ehrlichkeit. Die Planner-Konsum-Haelfte (:1000-1013 CONSUMES)
bleibt unangetastet -- sgl-project#1015-Terrain.
Bauform uebernommen von den beiden Instrumenten, die es im Baum richtig
machen: Total unabhaengig messen (torch.cuda.mem_get_info -> echtes CUDA-free,
dieselbe Quelle, der kv_backing_relief._free_bytes vertraut, nie
total-minus-used) PLUS expliziter Rest-Term (logindex flips-Form).

ERSTE ZEILE AM METALL (boot_855_r1026_..._102721.log):
  PP0 rest=13.875 measured free=16.888 unaccounted=+3.013
  PP1 rest= 8.440 measured free=12.057 unaccounted=+3.617
  PP2 rest= 7.898 measured free=10.405 unaccounted=+2.507

Das ist GENAU die sgl-project#1021-Capstone-Groesse, die die Uebergabe als "unchanged
and unused across all passes" fuehrt -- die Sizing-Zeile druckt sie ab jetzt
selbst, je Rang, auf jedem Boot. Die Reihenfolge des Koordinators (sgl-project#1026 VOR
sgl-project#1021) war damit richtig: sgl-project#1021 haette auf einer Zeile gerechnet, die diesen
Term nicht kannte.

SELBSTKORREKTUR im selben Zug: mein erster Kommentar deutete ein positives
Residuum als "KV-Pool ohne Gegenwert verschenkt". FALSCH. Der Ueberschuss ist
der UNGEBUCHTE Topf, aus dem die spaeteren Allokationen tatsaechlich bedient
werden -- die Graph-Capture (810 MiB) und die Arena-Tails kommen von genau
hier. Damit ist die Zahl die ehrliche Kopffreiheit und zugleich die ganze
sgl-project#1025-Geschichte in einer Groesse: gdncov bootet, weil hier 2,5-3,6 GiB
stehen (weit ueber den 810 MiB der Capture); das lm_head-Artefakt stirbt,
weil seine freigewordenen Gewichtsbytes in KV-Pool verwandelt werden und
GENAU DIESEN Ueberschuss unter den Capture-Bedarf druecken. Ein schrumpfendes
Residuum ueber Boots ist die Fruehwarnung, die vier Boots der sgl-project#855-Leiter
sich haben erarbeiten muessen. Kommentar entsprechend korrigiert.

ZUSTAND: Serving oben auf 32,16,16 + Makespan-Cut, health 200, greedy
"capital of France" -> " Paris.", Deadman armiert.
BELEG-STUFE: BOOT-GEMESSEN. Kein Suiten-Lauf (Speed-Modus); Abnahme-Suite
ist Posten 6.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…TIGT -- der Makespan-Cut bezahlt Balance mit Welt. Posten-2-Verdikt KORRIGIERT.

Einschub-Boot boot_855_arena3218_0840f82601_0830_103720.log, [32,18,14]/[8,4,4],
sonst byte-gleiche Flags, sgl-project#1026-Zeile und Regime-Instrument mitgefahren.
Verdikt-Regeln (a)-(d) des Koordinators vorab festgenagelt, hier abgearbeitet.

== (a) PRIMAERE LESEGROESSEN, [32,18,14] gegen [32,17,15] ==

  Groesse                    [32,18,14]      [32,17,15]     Delta
  PP2 Arena-Tail             1126.95 MiB     1493.22 MiB    -366 MiB (-24,5%)
  PP1 Arena-Tail              167.02 MiB        0.00 MiB
  PP0 Arena-Tail              436.45 MiB      436.45 MiB    unveraendert
  PP2 Pool (der Binder)        637.726         533.333      -16,4%
  Welt (min ueber Raenge)      618.778         533.333      -13,8%
  Bindender Rang               PP0             PP2          WECHSELT
  unaccounted wr0/1/2      +3.013/+3.684/+2.440   +3.013/+3.617/+2.507

== (b) VERDIKT: BESTAETIGT, beide Bedingungen erfuellt ==
PP2s Arena-Tail faellt materiell (-24,5%), UND die Welt ist trennbar: -13,8%
liegt ueber dem gemessenen ~10%-Welt-Varianzboden, und am BINDER ist es
eindeutig -- PP2s Pool ueber zwei Boots je Arm:
  [32,17,15]: 533.333 / 592.677   (Spanne 533-593k)
  [32,18,14]: 637.726 / 635.826   (Spanne 636-638k, 0,3% auseinander)
Die Spannen ueberlappen NICHT, und die [32,18,14]-Stichprobe ist eng. Der
Arena-Tail selbst ist ohnehin keine verrauschte Groesse (die Probe meldet
derived == measured, Fehler 0.00).

== (c) DER PREIS DER GEGENRICHTUNG, wie verlangt beziffert ==
Gleiche Ladder wie beim Makespan-Boot (74k/12k/50k), volle Chunks (4096):

  Rang   [32,18,14] n=32      [32,17,15] n=29
  PP0    13.271,8 ms           12.035,0 ms
  PP1    34.858,4 ms  Pacer    30.900,2 ms  Pacer
  PP2    29.869,7 ms           28.492,9 ms
  3080-Imbalance   16,70%              8,45%
  Pacer-Rate      3.760 tok/s         3.844 tok/s   (+2,2%)

Der Makespan-Cut HALBIERT die Imbalance-Kennzahl -- das war richtig gemessen.
Aber der Pacer gewinnt dabei nur 2,2%, und das liegt INNERHALB des
Decode-/Mess-Rauschens dieses Rigs. Die Kennzahl bewegt sich, der Durchsatz
nicht. Bezahlt werden dafuer 13,8% Welt.

== SELBSTKORREKTUR (vierte heute, und die folgenreichste) ==
Mein Posten-2-Commit acac781 verkuendete den Makespan-Cut als Gewinn "at
no capacity price", gestuetzt auf Welt 592.677 gegen 591.726 = +0,16%. Dieser
Vergleich war WERTLOS: er lag in einem Rauschen von ~10%, das ich selbst erst
danach gemessen habe (A/A-Paar), und er war zusaetzlich durch die
Seam-Record-Vererbung konfundiert. Sauber gegengetestet kostet der Cut 13,8%
Welt und 16,4% am Binder. Das Verdikt "Makespan-Cut ist ein Gewinn" ist
HIERMIT ZURUECKGEZOGEN. Es war ein Handel, und auf einem Kapazitaets-Strang
ist es derselbe schlechte Handel, den ich beim TP-Vektor-Arm 42,11,11
abgelehnt habe -- ich habe ihn dort erkannt und hier nicht, weil dort die
Zahl gross war und hier im Rauschen lag.

== EMPFEHLUNG FUER DIE SHIPPING-FORM: [32,18,14] ==
Serving steht bereits darauf (dieser Boot). Die Abnahme-Suite sollte auf der
Form laufen, die wir ausliefern -- also auf dieser, nicht auf dem Makespan-Cut.

== (d) MESSBASIS FUER sgl-project#1019 ==
Der Objective-Rebuild des Planner-Agenten bekommt hiermit einen zweiten
Datenpunkt, an dem BEIDE Objectives falsch liegen: makespan waehlt
[32,17,15] und optimiert eine Kennzahl, die sich nicht in Pacer-Durchsatz
uebersetzt, waehrend die Welt 13,8% faellt. Die fehlende Groesse im
Kostenmodell ist der ARENA-TAIL: er ist cut-abhaengig, deterministisch und
trifft genau den Rang, der die Welt bindet. Ich baue daran NICHT weiter.

BELEG-STUFE: BOOT-GEMESSEN, n=32 volle Chunks je Rang fuer Compute, n=2 Boots
je Arm fuer den Binder-Pool, n=1 je Arm fuer Arena-Tail (deterministische
Groesse). Kein Suiten-Lauf.
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
…ex (the same cuSOLVER potrf without the host info check) and an fp64 round-robin Jacobi eigh kernel (the fork's graph-safe decode solver widened to fp64, fixed sweeps, eigh's ascending order); whole-layer prefill commit graph back on for init_method=k31 (SGLANG_GDN_K31_EIGH=torch keeps the eager reference call); CPU tests vs torch.linalg.eigh (top-8 projector < 1e-12, all splits within the Davis-Kahan bound), GPU check (capture, bitwise replay, timing)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 27, 2026
…follow-up)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant