Repository navigation
Conversation
Contributor
Author
|
@zhyncs can you take a look at this PR, is there sth wrong with my understanding? It is sth that can affect our serving result. But if there is something wrong with my understanding, we can just close this PR. |
yichuan-w
marked this pull request as draft
August 10, 2024 17:58
yichuan-w
marked this pull request as ready for review
August 10, 2024 18:03
yichuan-w
marked this pull request as draft
August 10, 2024 18:11
Contributor
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-1.8B", trust_remote_code=True)
prompt = "Hello, how are you?"
token_ids_1 = tokenizer(prompt).input_ids
token_ids_2 = tokenizer.encode(prompt)
print("Using tokenizer():", token_ids_1)
print("Using tokenizer.encode():", token_ids_2)
print("Are the results identical?", token_ids_1 == token_ids_2)
print("\nSpecial tokens:")
print("tokenizer(): [BOS]:", tokenizer.bos_token_id in token_ids_1, ", [EOS]:", tokenizer.eos_token_id in token_ids_1)
print("encode(): [BOS]:", tokenizer.bos_token_id in token_ids_2, ", [EOS]:", tokenizer.eos_token_id in token_ids_2)
print("\nDecoded tokens:")
print("tokenizer():", tokenizer.decode(token_ids_1))
print("encode():", tokenizer.decode(token_ids_2)) |
Contributor
Author
|
Sorry, this problem does not arise from here, so I closed that, if needed maybe we can fix that in another PR. |
shun8686
pushed a commit
to shun8686/sglang
that referenced
this pull request
Jul 25, 2026
test_npu_openai_server to npu
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…rice the trade solve_pp_cut_for_kv_floor and world_kv_floor have sat in planner/pp_cut.py since sgl-project#485 with no caller outside that file -- built, proven, and invisible to every boot. --pp-solve-cut hard-wired the MAKESPAN solver, so a rig whose operator is pushing capacity got a cut optimized for prefill speed and no way to say otherwise. This is the built-never-wired class, and the fix is wiring, not new solver code: both solvers already existed and neither is modified. --pp-solve-objective kv-floor|makespan dispatches between them. DEFAULT is kv-floor, because the standing boot is a capacity push; makespan stays selectable. THE TRADE IS NOW A NUMBER IN THE BOOT LOG. Whichever objective is chosen, the provenance line prices the chosen cut AND the cut the other objective would have returned, both on BOTH axes (pool tokens and makespan ms). Measured end-to-end on census-855-v2 at this operating point: objective=kv-floor chose [31,18,15] pool 268494 tokens makespan 3662.1 ms objective=makespan chose [32,17,15] pool 265213 tokens makespan 3089.6 ms so on THIS config kv-floor buys +3281 tokens (+1.24 %) for +572.5 ms (+18.5 %). That is a worse trade than the -3.9 % pool figure the makespan cut was first reported against, and it is exactly why the line exists: the operator can now see the price per boot and pick, instead of inferring it afterwards from a pool number that moved. The [32,17,15] / makespan=3089.6 ms result reproduces the reported boot exactly. The comparison solve is a pure desk computation over the same inputs and is wrapped so it can never break a boot: a failure there degrades the LINE, not the verdict. The chosen cut's own feasibility is unchanged and still refuses by name. Also note which cut carries the corridor dip: [32,17,15] emits the #1009a dip line on stage0, [31,18,15] does not, because it leaves a layer off the 5090. That is visible per objective now rather than discovered on metal. DESK-PROVEN end to end through ServerArgs for both objectives: pp_layer_ratio resolves to [31,18,15] and [32,17,15] respectively, the trade line prints both cuts on both axes in both directions, and the dip line fires only for the cut that earns it. ruff unchanged (358). No solver logic touched.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…ool ranking INVERTS Merged 8703bc0 and booted --pp-solve-objective kv-floor. BOOT-PROVEN: up, coherent, deadman armed. Numbers only, no objective verdict -- the fork is the user's. SOLVED CUT: --pp-layer-ratio 31,18,15, attention per stage [7,5,4] of 16, makespan 3662.1 ms, pacer=stage1, min_headroom=62 MiB. Provenance line printed both objectives as designed. (a) PP WORLD TOKENS ON METAL (the min over ranks, which is what serves): cut attn PP0 PP1 PP2 PP WORLD (min) [32,18,14] pin [8,4,4] 616670 725191 640066 616,670 [32,17,15] msp [8,4,4] 643198 729329 592677 592,677 [31,18,15] kvf [7,5,4] 674737 567412 592677 567,412 kv-floor is the WORST of the three: -49,258 vs the hand pin (-8.0 %) and -25,265 vs the makespan cut (-4.3 %). THE SOLVER PREDICTED THE OPPOSITE DIRECTION: kv-floor 268,494 tokens vs makespan 265,213 (+1.24 % for kv-floor). On metal makespan beats kv-floor by +4.5 %. The ranking INVERTS. Indicator law: the solver's pool metric is not measuring the quantity that serves. It is not the PP world min -- the world min is bounded per rank by available_bytes / (n_attn_layers * 2048), so the ATTENTION split dominates it, and kv-floor moved an attention layer onto PP1 ([8,4,4] -> [7,5,4]), whose capacity then fell 725,191 -> 567,412. This is exactly what ANALYSE_799 section 5.4 predicted for this rig: on the current budgets PP0 is no longer the rich stage, so shifting attention layers makes the min WORSE, and the historic "+33 %" projection inverts. The kv-floor objective walked straight into it. Neither solved cut beats the hand pin on capacity. (b) MEASURED PREFILL, same ladder discipline (unique prompts, cached_tokens=0 verified, one request at a time, first post-boot sample discarded as the systematic cold start): cut 12k 50k (two passes) 50k mean [32,18,14] pin 3.831 s 14.753 / 15.301 s 15.03 s [31,18,15] kvf 3.780 s 15.527 / 16.092 s 15.81 s kv-floor is +5.2 % slower at 50k against the hand pin; 12k is inside noise. The solver's +18.5 % makespan penalty does NOT reproduce at that magnitude end to end -- same dilution this strand already measured when the microbench's 1.39-1.46x layer-level prefill gain arrived as 1.05-1.21x e2e. Treat the makespan axis as a RANKING signal, not a wall-clock prediction. (c) CORRIDOR / DIP: kv-floor emits ZERO corridor-dip lines, as predicted -- [31,18,15] keeps a layer off the 5090 and does not run stage0 at the band edge. The makespan cut [32,17,15] emitted one (157.0 MiB above at-rest, 48.0 of the 205 MiB allowance). SO THE FORK, stated as three numbers and no recommendation: hand pin [32,18,14] 616,670 tok 15.03 s @50k dip: n/a (not solved) makespan [32,17,15] 592,677 tok not measured dip: 1 line on stage0 kv-floor [31,18,15] 567,412 tok 15.81 s @50k dip: none The capacity push wanted the kv-floor objective. On this rig, at this operating point, it costs capacity AND prefill against the pin it replaced. Serving stays on it per the standing order; no rollback to hand pins.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…makespan, record why My sgl-project#1018 default was wrong and metal caught it. Three cuts, PP world tokens = min_r(available_r / (n_attn_r * 2048)): hand-pin [32,18,14] 616670 makespan [32,17,15] 592677 kv-floor [31,18,15] 567412 The solver had predicted kv-floor ABOVE makespan. The ranking INVERTED, and both solved cuts lost to the hand pin. Default is now makespan; kv-floor stays selectable for experiments and WARNS when selected. THE "IT SUMS INSTEAD OF TAKING THE MIN" HYPOTHESIS IS FALSE. world_kv_floor reduces with min(floor, cap). The defect is the QUANTITY reduced, not the reduction, and I checked before repeating the guess: * stage_kv_capacity charges the DUAL-PHASE arena, max(PP attention share, TP token share), because one arena backs both flip layouts. On this rig that bills stage1 for 4.75 attention-equivalents where the PP boot bills 4. * its spendable pot is the CUT GATE's -- what survives corridor, transient, fixed overhead and seam. Right for "does this cut fit", wrong for "how many tokens will the boot address": 2.33x below the boot's available bytes on the same cut and stage (my 265213 against metal 616670 on stage0). * it therefore cannot separate cuts sharing a binding stage: it returned the IDENTICAL 265213.33 for [32,18,14] and [32,17,15], which metal separates by 23993 tokens. PREDICTED BEFORE IT WAS BUILT, and the reference now lives in the docstrings so the next reader cannot miss it as I did: docs/dev/ANALYSE_799_token_axis_x_pp.md section 5.4 measured that on the CURRENT budgets PP0 is the POOR stage, so moving attention off it makes the min worse -- the historic "+33 % for [6,5,5]" inverts to -16 %. kv-floor moved an attention layer onto PP1 ([8,4,4] -> [7,5,4]) and PP1 fell 725191 -> 567412. That section also caps the token axis at 639800 (+6.51 %) at this cut. world_kv_floor is not deleted: it remains correct for the feasibility-scoped comparison inside the cut gate, which is what it was built for. Its docstring now says, in its own words, do not rank serving cuts with it. MEASURED SIDE-FINDING, and it is larger than the objective question. The kv-floor boot reports PP1 3.44 GB and PP2 2.12 GB available_gpu_mem -- 3280.6 and 2021.8 MiB, far ABOVE the 819-1229 MiB corridor band, which that law scores as a failed acceptance in its own right. Converting that unbuilt free at the poor ranks, which are exactly the ranks that set the world min: target free PP1 gain PP2 gain world min vs hand pin 1229 MiB +210088 +101477 674737 +58067 (+9.42 %) 1024 MiB +231080 +127717 674737 +58067 (+9.42 %) 819 MiB +252072 +153957 674737 +9.42 % The result is INSENSITIVE to the band target because PP0 becomes the binding rank in all three cases at 674737 -- its own capacity is untouched. So the prize is +9.42 % over the hand pin, which alone exceeds ANALYSE_799's +6.51 % ceiling; that ceiling held budgets fixed, and this is a different lever. AND IT REVERSES THE LESSON ABOUT kv-floor's DIRECTION. Once the free is converted PP0 binds, and the right move is then to shift attention OFF PP0 -- which is what kv-floor tried to do. kv-floor was not wrong in principle; it optimized for a state this boot does not currently reach because the free is never built. That is worth more than re-ranking cuts against today's state. Unit caveat, stated rather than smoothed: 3.44/2.12 were logged as "GB" and read here as decimal 1e9. If they are GiB the excess is larger and every conclusion above strengthens. Also available_gpu_mem is the CUDA runtime's free, not NVML's, and the corridor law is written on NVML free -- the ~424/518 MiB carve-out sits between them. DESK-PROVEN: default resolves to makespan -> [32,17,15]; kv-floor still resolves to [31,18,15] and emits the falsification warning; the trade line still prints both cuts on both axes. ruff unchanged (359).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…s ~2x and breaks serving d7c3736 (sgl-project#1015) was merged, booted, and REVERTED (a204832). It is not a mis-scoped refactor -- it takes serving down. MEASURED, boot_855_cons1_0840f82601_0830_073758 vs the same tree without it: TP-phase "weights + runtime state" (GiB) before sgl-project#1015 with sgl-project#1015 PP0 4.252 26.581 PP1 2.719 14.788 PP2 3.662 15.435 sum 10.633 56.804 The checkpoint is 27.5 GiB. sgl-project#1015 swung the post from under-booking (10.63, the defect I reported in sgl-project#1014) straight past correct to ~2x OVER-booking: 56.8 GiB of weights charged for a 27.5 GiB model, i.e. roughly the WHOLE model on EVERY rank instead of that rank's shard. Consequence, same boot: TP-phase PP0 KV budget "rest" 24.195 GiB -> 1.867 GiB EFFECTIVE max_total_num_tokens 615,434 -> 185,344 (-70 %) per-rank capacities 597210/448290/449378 -> 61152/57920/68724 coherence probe (/generate, greedy, "capital of France") -> NO RESPONSE The recalibrator then reacted to the collapsed capacities and moved the token vector [30,17,17] -> [21,20,23], which is correct behaviour on wrong inputs and would have written a garbage vector into the next boot's hint. REVERT VERIFIED: a204832 restores TP world 615,434 and the probe answers "Paris". One file, 60 lines (model_executor/model_runner_kv_cache_mixin.py) -- so the runtime blast radius of d7c3736 is exactly the weight post, which isolates the attribution. WHAT I GOT WRONG AND SHOULD HAVE CAUGHT FIRST: I supplied the sgl-project#1014 diff that motivated this fix, including the expected TP shard figures (~13.75/6.9/6.9 GiB from the 32,16,16 vector). I flagged at the time that I had NOT verified whether the flip re-materialises weights or re-views the PP residency, and that if it re-views them the "correct" TP post may legitimately be smaller than shard arithmetic. That caveat turned out to be the whole question, and a fix built on the arithmetic alone overshot. A boot-time falsifier -- assert the summed TP weight post lies between the largest single rank shard and the whole checkpoint -- would have caught this at the desk. The rest of the planner chain (#1009a/sgl-project#1010/sgl-project#1018/sgl-project#1019) is NOT reverted: it came in via cc70a6d and the earlier merges and is unaffected. ALSO ON THIS BOOT, and it works: sgl-project#778 cold-post-spill (5379340) is live and its markers fire on the real seam -- "phase-release LEND pp_to_tp: 65536 rows (~1024 MiB)" on PP0, 131072 rows (~1024 MiB) on PP1 and PP2 (row counts differ because the PP cell differs: 16384 B on PP0, 8192 B on PP1/PP2), "phase-release REPAY tp_to_pp: loan -> 0 rows" on all three before the prefill leg resumes. REPAY reports "0 MiB returned" -- the dial had not yet grown into the credit at that seam, so the repayment had nothing to hand back. That is the convergence behaviour the author described, not a leak, but it means the LEND's token effect is NOT yet demonstrated on metal and must not be counted in any ladder until a boot under load shows the pool actually taking it.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…anchor, and measure the loss KV is DIVISIBLE, the recurrent (mamba/GDN) state is POINTLIKE. Every length-shifting operation in the load-back chain is KV-built, so the extent it produces lands, for mamba, on a boundary with no state behind it -- and reusing a KV prefix there resumes a scan from a state that consumed different tokens. That is silently wrong, and today it is answered ALL-OR-NOTHING: the whole load-back falls and the whole prompt re-prefills. The user's grant is the opposite: re-prefill at most one HiCache chunk, from the KV position that matches the most current GDN state. WHAT THIS COMMIT DOES * Publishes the anchor depth the match walk already computed but discarded. `MatchResult.state_anchor_depth` is the absolute depth of `best_match_node`, i.e. of the deepest node EVERY component accepted -- mamba's acceptor being `is_resume_candidate` itself. The predicate is REUSED, never re-derived; a second anchor rule would be the second bookkeeping the upstream-minimal law rejects, and sgl-project#747 records what two anchor lineages do to each other. * `MatchResult.key_match_depth` beside it: how far the KEY matched with no validator asked. The PAIR is what separates "the prefix was never stored" from "the prefix is here and its anchor died with an evicted node" (sgl-project#1039). One hit number collapses them. * `state_aligned_load_back_len()` -- ONE writer, called by BOTH PPAdmissionEntry constructors, rounds PP0's extent DOWN to the anchor and counts what it gave back. `None` (no state-bearing component) keeps upstream byte-for-byte; `0` (no anchor on this path) is the CORRECT "load back nothing" under leaf-only mamba data, never an error. * `zero_match_result` zeroes the anchor depth too -- a refusal that reaches back to the root has no anchor left to point at. Same contract as sgl-project#1018 fixed for `cache_protected_len`. THREE VERIFICATIONS THAT CHANGED THE BUILD (desk-proven at this tree) * mamba_component.py's `best_depth` is NOT the number to plumb: its block needs `SGLANG_MAMBA_CKPT_STRICT_RESUME` (default False) AND `cache_controller is None` (false under HiCache), and it sums DEVICE value chunks, which diverge from the accepted-anchor depth exactly when `separate_device_match` is on. * The extent is ALREADY state-aligned RANK-LOCALLY: `host_hit_length` is measured from `best_match_node` (full_component.py), and that node is only set where the mamba validator accepted. A rank-local rounding is therefore a no-op, and is not built as one. * The real misalignment is CROSS-RANK, at the S1 clamp: PP0's extent is aligned in PP0's tree, the cut lands in the receiving rank's. That seam is MEASURED here (the FIX-3 refusal now prints the local anchor depth and whether it AGREEs with the told extent), not yet fixed -- the population decides the design. INSTRUMENTS (each names its denominator) * `sgl-project#1040 EXTENT STATE-ALIGN`: kv / extent / loss / anchor_depth / device_len / key_match_depth / class, with running totals. The round-down loss is kept APART from tokens forgone because no anchor existed -- pooling them inflated loss_max from 200 to 300 on the check fixture, which is how it was caught. * `sgl-project#988 LOADBACK` gains the KV/MAMBA split: kv_applied vs mamba_restored vs kv_only. KV moved while the restore refused is a HALF SUCCESS with a named link, never a green one. * `sgl-project#968 LOAD-BACK ROW APPLIED` gains `anchors=` and `retract_survivors=`. * `reset_for_retract` stamps retraction survival -- the free pre-measurement for the lifecycle half (sgl-project#1039); monotone, never cleared, nothing stale behind it. CHECK (speed mode: one matched check, not a battery) devtools/check_1040_rounding.py -- hermetic, CUDA-free. Named failure class of this edit: unit/coordinate error (absolute anchor depth vs the row's delta), off-by-chunk at the boundary, empty candidate set treated as an error. Trips the rounding on empty / below / exactly-at / above / pure-KV / no-hit, and walks `_state_anchor_depth` over a stub chain to pin the unit. ALL PASS, after it failed on the pooled loss denominator and that was fixed. py_compile + import smoke on all seven files; ruff clean on the three most-edited. cell_1039_ingeneration.sh: stage (iii) moved off `sgl-project#968 LOAD-BACK PROMOTED`, which the row collapse DELETED (0 live emitters, only a prose mention left -- the sgl-project#995 trap, it would have read 0 forever), onto `sgl-project#968 LOAD-BACK ROW APPLIED`. Adds the KV/mamba split, the measured loss median/max, the anchor-absent population, and a zombie test over the four abolished markers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
I found the len of ids is different from which I print from tp_worder and which we preprocess in python/sglang/bench_serving.py, and I found there is some difference between prompt_token_ids = tokenizer.encode(prompt) and prompt_token_ids = tokenizer(prompt).input_ids so I change the prompt_token_ids = tokenizer(prompt).input_ids to align with the tokenizer_manager
Modification
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.