Skip to content

Fix tokenizer in serving - #1018

Closed
yichuan-w wants to merge 1 commit into
sgl-project:mainfrom
yichuan-w:fix_serving_tokenizer
Closed

yichuan-w wants to merge 1 commit into
sgl-project:mainfrom
yichuan-w:fix_serving_tokenizer

Conversation

@yichuan-w

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

I found the len of ids is different from which I print from tp_worder and which we preprocess in python/sglang/bench_serving.py, and I found there is some difference between prompt_token_ids = tokenizer.encode(prompt) and prompt_token_ids = tokenizer(prompt).input_ids so I change the prompt_token_ids = tokenizer(prompt).input_ids to align with the tokenizer_manager

Modification

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@yichuan-w
yichuan-w requested a review from zhyncs August 10, 2024 17:44
@yichuan-w

Copy link
Copy Markdown
Contributor Author

@zhyncs can you take a look at this PR, is there sth wrong with my understanding? It is sth that can affect our serving result. But if there is something wrong with my understanding, we can just close this PR.

@yichuan-w
yichuan-w marked this pull request as draft August 10, 2024 17:58
@yichuan-w
yichuan-w marked this pull request as ready for review August 10, 2024 18:03
@yichuan-w
yichuan-w marked this pull request as draft August 10, 2024 18:11
@yichuan-w yichuan-w closed this Aug 10, 2024
@zhyncs

zhyncs commented Aug 10, 2024

Copy link
Copy Markdown
Contributor
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-1.8B", trust_remote_code=True)

prompt = "Hello, how are you?"

token_ids_1 = tokenizer(prompt).input_ids
token_ids_2 = tokenizer.encode(prompt)

print("Using tokenizer():", token_ids_1)
print("Using tokenizer.encode():", token_ids_2)

print("Are the results identical?", token_ids_1 == token_ids_2)

print("\nSpecial tokens:")
print("tokenizer(): [BOS]:", tokenizer.bos_token_id in token_ids_1, ", [EOS]:", tokenizer.eos_token_id in token_ids_1)
print("encode(): [BOS]:", tokenizer.bos_token_id in token_ids_2, ", [EOS]:", tokenizer.eos_token_id in token_ids_2)

print("\nDecoded tokens:")
print("tokenizer():", tokenizer.decode(token_ids_1))
print("encode():", tokenizer.decode(token_ids_2))
Using tokenizer(): [9707, 11, 1246, 525, 498, 30]
Using tokenizer.encode(): [9707, 11, 1246, 525, 498, 30]
Are the results identical? True

Special tokens:
tokenizer(): [BOS]: False , [EOS]: False
encode(): [BOS]: False , [EOS]: False

Decoded tokens:
tokenizer(): Hello, how are you?
encode(): Hello, how are you?

@yichuan-w

Copy link
Copy Markdown
Contributor Author

Sorry, this problem does not arise from here, so I closed that, if needed maybe we can fix that in another PR.

shun8686 pushed a commit to shun8686/sglang that referenced this pull request Jul 25, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…rice the trade

solve_pp_cut_for_kv_floor and world_kv_floor have sat in planner/pp_cut.py
since sgl-project#485 with no caller outside that file -- built, proven, and invisible
to every boot. --pp-solve-cut hard-wired the MAKESPAN solver, so a rig whose
operator is pushing capacity got a cut optimized for prefill speed and no way
to say otherwise. This is the built-never-wired class, and the fix is wiring,
not new solver code: both solvers already existed and neither is modified.

--pp-solve-objective kv-floor|makespan dispatches between them. DEFAULT is
kv-floor, because the standing boot is a capacity push; makespan stays
selectable.

THE TRADE IS NOW A NUMBER IN THE BOOT LOG. Whichever objective is chosen, the
provenance line prices the chosen cut AND the cut the other objective would
have returned, both on BOTH axes (pool tokens and makespan ms). Measured
end-to-end on census-855-v2 at this operating point:

  objective=kv-floor  chose [31,18,15]  pool 268494 tokens  makespan 3662.1 ms
  objective=makespan  chose [32,17,15]  pool 265213 tokens  makespan 3089.6 ms

so on THIS config kv-floor buys +3281 tokens (+1.24 %) for +572.5 ms
(+18.5 %). That is a worse trade than the -3.9 % pool figure the makespan cut
was first reported against, and it is exactly why the line exists: the
operator can now see the price per boot and pick, instead of inferring it
afterwards from a pool number that moved. The [32,17,15] / makespan=3089.6 ms
result reproduces the reported boot exactly.

The comparison solve is a pure desk computation over the same inputs and is
wrapped so it can never break a boot: a failure there degrades the LINE, not
the verdict. The chosen cut's own feasibility is unchanged and still refuses
by name.

Also note which cut carries the corridor dip: [32,17,15] emits the #1009a dip
line on stage0, [31,18,15] does not, because it leaves a layer off the 5090.
That is visible per objective now rather than discovered on metal.

DESK-PROVEN end to end through ServerArgs for both objectives:
pp_layer_ratio resolves to [31,18,15] and [32,17,15] respectively, the trade
line prints both cuts on both axes in both directions, and the dip line fires
only for the cut that earns it. ruff unchanged (358). No solver logic touched.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…ool ranking INVERTS

Merged 8703bc0 and booted --pp-solve-objective kv-floor. BOOT-PROVEN: up,
coherent, deadman armed. Numbers only, no objective verdict -- the fork is the
user's.

SOLVED CUT: --pp-layer-ratio 31,18,15, attention per stage [7,5,4] of 16,
makespan 3662.1 ms, pacer=stage1, min_headroom=62 MiB. Provenance line printed
both objectives as designed.

(a) PP WORLD TOKENS ON METAL (the min over ranks, which is what serves):

  cut              attn      PP0      PP1      PP2    PP WORLD (min)
  [32,18,14] pin  [8,4,4] 616670   725191   640066      616,670
  [32,17,15] msp  [8,4,4] 643198   729329   592677      592,677
  [31,18,15] kvf  [7,5,4] 674737   567412   592677      567,412

  kv-floor is the WORST of the three: -49,258 vs the hand pin (-8.0 %) and
  -25,265 vs the makespan cut (-4.3 %).

  THE SOLVER PREDICTED THE OPPOSITE DIRECTION: kv-floor 268,494 tokens vs
  makespan 265,213 (+1.24 % for kv-floor). On metal makespan beats kv-floor by
  +4.5 %. The ranking INVERTS. Indicator law: the solver's pool metric is not
  measuring the quantity that serves. It is not the PP world min -- the world
  min is bounded per rank by available_bytes / (n_attn_layers * 2048), so the
  ATTENTION split dominates it, and kv-floor moved an attention layer onto
  PP1 ([8,4,4] -> [7,5,4]), whose capacity then fell 725,191 -> 567,412.

  This is exactly what ANALYSE_799 section 5.4 predicted for this rig: on the
  current budgets PP0 is no longer the rich stage, so shifting attention
  layers makes the min WORSE, and the historic "+33 %" projection inverts. The
  kv-floor objective walked straight into it. Neither solved cut beats the
  hand pin on capacity.

(b) MEASURED PREFILL, same ladder discipline (unique prompts, cached_tokens=0
    verified, one request at a time, first post-boot sample discarded as the
    systematic cold start):

  cut               12k            50k (two passes)      50k mean
  [32,18,14] pin   3.831 s      14.753 / 15.301 s        15.03 s
  [31,18,15] kvf   3.780 s      15.527 / 16.092 s        15.81 s

  kv-floor is +5.2 % slower at 50k against the hand pin; 12k is inside noise.
  The solver's +18.5 % makespan penalty does NOT reproduce at that magnitude
  end to end -- same dilution this strand already measured when the microbench's
  1.39-1.46x layer-level prefill gain arrived as 1.05-1.21x e2e. Treat the
  makespan axis as a RANKING signal, not a wall-clock prediction.

(c) CORRIDOR / DIP: kv-floor emits ZERO corridor-dip lines, as predicted --
  [31,18,15] keeps a layer off the 5090 and does not run stage0 at the band
  edge. The makespan cut [32,17,15] emitted one (157.0 MiB above at-rest, 48.0
  of the 205 MiB allowance).

SO THE FORK, stated as three numbers and no recommendation:
  hand pin [32,18,14]  616,670 tok  15.03 s @50k  dip: n/a (not solved)
  makespan [32,17,15]  592,677 tok  not measured   dip: 1 line on stage0
  kv-floor [31,18,15]  567,412 tok  15.81 s @50k   dip: none

The capacity push wanted the kv-floor objective. On this rig, at this
operating point, it costs capacity AND prefill against the pin it replaced.
Serving stays on it per the standing order; no rollback to hand pins.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…makespan, record why

My sgl-project#1018 default was wrong and metal caught it. Three cuts, PP world tokens
= min_r(available_r / (n_attn_r * 2048)):

  hand-pin [32,18,14]  616670
  makespan [32,17,15]  592677
  kv-floor [31,18,15]  567412

The solver had predicted kv-floor ABOVE makespan. The ranking INVERTED, and
both solved cuts lost to the hand pin. Default is now makespan; kv-floor stays
selectable for experiments and WARNS when selected.

THE "IT SUMS INSTEAD OF TAKING THE MIN" HYPOTHESIS IS FALSE. world_kv_floor
reduces with min(floor, cap). The defect is the QUANTITY reduced, not the
reduction, and I checked before repeating the guess:

* stage_kv_capacity charges the DUAL-PHASE arena, max(PP attention share, TP
  token share), because one arena backs both flip layouts. On this rig that
  bills stage1 for 4.75 attention-equivalents where the PP boot bills 4.
* its spendable pot is the CUT GATE's -- what survives corridor, transient,
  fixed overhead and seam. Right for "does this cut fit", wrong for "how many
  tokens will the boot address": 2.33x below the boot's available bytes on the
  same cut and stage (my 265213 against metal 616670 on stage0).
* it therefore cannot separate cuts sharing a binding stage: it returned the
  IDENTICAL 265213.33 for [32,18,14] and [32,17,15], which metal separates by
  23993 tokens.

PREDICTED BEFORE IT WAS BUILT, and the reference now lives in the docstrings
so the next reader cannot miss it as I did:
docs/dev/ANALYSE_799_token_axis_x_pp.md section 5.4 measured that on the
CURRENT budgets PP0 is the POOR stage, so moving attention off it makes the
min worse -- the historic "+33 % for [6,5,5]" inverts to -16 %. kv-floor moved
an attention layer onto PP1 ([8,4,4] -> [7,5,4]) and PP1 fell 725191 ->
567412. That section also caps the token axis at 639800 (+6.51 %) at this cut.

world_kv_floor is not deleted: it remains correct for the feasibility-scoped
comparison inside the cut gate, which is what it was built for. Its docstring
now says, in its own words, do not rank serving cuts with it.

MEASURED SIDE-FINDING, and it is larger than the objective question.
The kv-floor boot reports PP1 3.44 GB and PP2 2.12 GB available_gpu_mem --
3280.6 and 2021.8 MiB, far ABOVE the 819-1229 MiB corridor band, which that
law scores as a failed acceptance in its own right. Converting that unbuilt
free at the poor ranks, which are exactly the ranks that set the world min:

  target free   PP1 gain      PP2 gain      world min   vs hand pin
  1229 MiB      +210088       +101477       674737      +58067  (+9.42 %)
  1024 MiB      +231080       +127717       674737      +58067  (+9.42 %)
   819 MiB      +252072       +153957       674737      +9.42 %

The result is INSENSITIVE to the band target because PP0 becomes the binding
rank in all three cases at 674737 -- its own capacity is untouched. So the
prize is +9.42 % over the hand pin, which alone exceeds ANALYSE_799's +6.51 %
ceiling; that ceiling held budgets fixed, and this is a different lever.

AND IT REVERSES THE LESSON ABOUT kv-floor's DIRECTION. Once the free is
converted PP0 binds, and the right move is then to shift attention OFF PP0 --
which is what kv-floor tried to do. kv-floor was not wrong in principle; it
optimized for a state this boot does not currently reach because the free is
never built. That is worth more than re-ranking cuts against today's state.

Unit caveat, stated rather than smoothed: 3.44/2.12 were logged as "GB" and
read here as decimal 1e9. If they are GiB the excess is larger and every
conclusion above strengthens. Also available_gpu_mem is the CUDA runtime's
free, not NVML's, and the corridor law is written on NVML free -- the ~424/518
MiB carve-out sits between them.

DESK-PROVEN: default resolves to makespan -> [32,17,15]; kv-floor still
resolves to [31,18,15] and emits the falsification warning; the trade line
still prints both cuts on both axes. ruff unchanged (359).
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…s ~2x and breaks serving

d7c3736 (sgl-project#1015) was merged, booted, and REVERTED (a204832). It is not a
mis-scoped refactor -- it takes serving down.

MEASURED, boot_855_cons1_0840f82601_0830_073758 vs the same tree without it:

  TP-phase "weights + runtime state" (GiB)   before sgl-project#1015   with sgl-project#1015
    PP0                                          4.252        26.581
    PP1                                          2.719        14.788
    PP2                                          3.662        15.435
    sum                                         10.633        56.804

  The checkpoint is 27.5 GiB. sgl-project#1015 swung the post from under-booking (10.63,
  the defect I reported in sgl-project#1014) straight past correct to ~2x OVER-booking:
  56.8 GiB of weights charged for a 27.5 GiB model, i.e. roughly the WHOLE
  model on EVERY rank instead of that rank's shard.

  Consequence, same boot:
    TP-phase PP0 KV budget "rest"    24.195 GiB  ->   1.867 GiB
    EFFECTIVE max_total_num_tokens    615,434    ->   185,344   (-70 %)
    per-rank capacities         597210/448290/449378 -> 61152/57920/68724
    coherence probe (/generate, greedy, "capital of France")  -> NO RESPONSE

  The recalibrator then reacted to the collapsed capacities and moved the
  token vector [30,17,17] -> [21,20,23], which is correct behaviour on wrong
  inputs and would have written a garbage vector into the next boot's hint.

REVERT VERIFIED: a204832 restores TP world 615,434 and the probe answers
"Paris". One file, 60 lines
(model_executor/model_runner_kv_cache_mixin.py) -- so the runtime blast radius
of d7c3736 is exactly the weight post, which isolates the attribution.

WHAT I GOT WRONG AND SHOULD HAVE CAUGHT FIRST: I supplied the sgl-project#1014 diff that
motivated this fix, including the expected TP shard figures (~13.75/6.9/6.9
GiB from the 32,16,16 vector). I flagged at the time that I had NOT verified
whether the flip re-materialises weights or re-views the PP residency, and
that if it re-views them the "correct" TP post may legitimately be smaller
than shard arithmetic. That caveat turned out to be the whole question, and a
fix built on the arithmetic alone overshot. A boot-time falsifier -- assert
the summed TP weight post lies between the largest single rank shard and the
whole checkpoint -- would have caught this at the desk.

The rest of the planner chain (#1009a/sgl-project#1010/sgl-project#1018/sgl-project#1019) is NOT reverted: it
came in via cc70a6d and the earlier merges and is unaffected.

ALSO ON THIS BOOT, and it works: sgl-project#778 cold-post-spill (5379340) is live and
its markers fire on the real seam --
  "phase-release LEND pp_to_tp: 65536 rows (~1024 MiB)" on PP0,
  131072 rows (~1024 MiB) on PP1 and PP2 (row counts differ because the PP
  cell differs: 16384 B on PP0, 8192 B on PP1/PP2),
  "phase-release REPAY tp_to_pp: loan -> 0 rows" on all three before the
  prefill leg resumes.
REPAY reports "0 MiB returned" -- the dial had not yet grown into the credit
at that seam, so the repayment had nothing to hand back. That is the
convergence behaviour the author described, not a leak, but it means the
LEND's token effect is NOT yet demonstrated on metal and must not be counted
in any ladder until a boot under load shows the pool actually taking it.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 31, 2026
…anchor, and measure the loss

KV is DIVISIBLE, the recurrent (mamba/GDN) state is POINTLIKE. Every
length-shifting operation in the load-back chain is KV-built, so the extent it
produces lands, for mamba, on a boundary with no state behind it -- and reusing
a KV prefix there resumes a scan from a state that consumed different tokens.
That is silently wrong, and today it is answered ALL-OR-NOTHING: the whole
load-back falls and the whole prompt re-prefills. The user's grant is the
opposite: re-prefill at most one HiCache chunk, from the KV position that
matches the most current GDN state.

WHAT THIS COMMIT DOES
* Publishes the anchor depth the match walk already computed but discarded.
  `MatchResult.state_anchor_depth` is the absolute depth of `best_match_node`,
  i.e. of the deepest node EVERY component accepted -- mamba's acceptor being
  `is_resume_candidate` itself. The predicate is REUSED, never re-derived; a
  second anchor rule would be the second bookkeeping the upstream-minimal law
  rejects, and sgl-project#747 records what two anchor lineages do to each other.
* `MatchResult.key_match_depth` beside it: how far the KEY matched with no
  validator asked. The PAIR is what separates "the prefix was never stored"
  from "the prefix is here and its anchor died with an evicted node" (sgl-project#1039).
  One hit number collapses them.
* `state_aligned_load_back_len()` -- ONE writer, called by BOTH PPAdmissionEntry
  constructors, rounds PP0's extent DOWN to the anchor and counts what it gave
  back. `None` (no state-bearing component) keeps upstream byte-for-byte; `0`
  (no anchor on this path) is the CORRECT "load back nothing" under leaf-only
  mamba data, never an error.
* `zero_match_result` zeroes the anchor depth too -- a refusal that reaches back
  to the root has no anchor left to point at. Same contract as sgl-project#1018 fixed for
  `cache_protected_len`.

THREE VERIFICATIONS THAT CHANGED THE BUILD (desk-proven at this tree)
* mamba_component.py's `best_depth` is NOT the number to plumb: its block needs
  `SGLANG_MAMBA_CKPT_STRICT_RESUME` (default False) AND `cache_controller is
  None` (false under HiCache), and it sums DEVICE value chunks, which diverge
  from the accepted-anchor depth exactly when `separate_device_match` is on.
* The extent is ALREADY state-aligned RANK-LOCALLY: `host_hit_length` is
  measured from `best_match_node` (full_component.py), and that node is only set
  where the mamba validator accepted. A rank-local rounding is therefore a
  no-op, and is not built as one.
* The real misalignment is CROSS-RANK, at the S1 clamp: PP0's extent is aligned
  in PP0's tree, the cut lands in the receiving rank's. That seam is MEASURED
  here (the FIX-3 refusal now prints the local anchor depth and whether it
  AGREEs with the told extent), not yet fixed -- the population decides the
  design.

INSTRUMENTS (each names its denominator)
* `sgl-project#1040 EXTENT STATE-ALIGN`: kv / extent / loss / anchor_depth / device_len /
  key_match_depth / class, with running totals. The round-down loss is kept
  APART from tokens forgone because no anchor existed -- pooling them inflated
  loss_max from 200 to 300 on the check fixture, which is how it was caught.
* `sgl-project#988 LOADBACK` gains the KV/MAMBA split: kv_applied vs mamba_restored vs
  kv_only. KV moved while the restore refused is a HALF SUCCESS with a named
  link, never a green one.
* `sgl-project#968 LOAD-BACK ROW APPLIED` gains `anchors=` and `retract_survivors=`.
* `reset_for_retract` stamps retraction survival -- the free pre-measurement for
  the lifecycle half (sgl-project#1039); monotone, never cleared, nothing stale behind it.

CHECK (speed mode: one matched check, not a battery)
devtools/check_1040_rounding.py -- hermetic, CUDA-free. Named failure class of
this edit: unit/coordinate error (absolute anchor depth vs the row's delta),
off-by-chunk at the boundary, empty candidate set treated as an error. Trips the
rounding on empty / below / exactly-at / above / pure-KV / no-hit, and walks
`_state_anchor_depth` over a stub chain to pin the unit. ALL PASS, after it
failed on the pooled loss denominator and that was fixed.
py_compile + import smoke on all seven files; ruff clean on the three
most-edited.

cell_1039_ingeneration.sh: stage (iii) moved off `sgl-project#968 LOAD-BACK PROMOTED`,
which the row collapse DELETED (0 live emitters, only a prose mention left --
the sgl-project#995 trap, it would have read 0 forever), onto `sgl-project#968 LOAD-BACK ROW APPLIED`.
Adds the KV/mamba split, the measured loss median/max, the anchor-absent
population, and a zombie test over the four abolished markers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants