Repository navigation
Update supported models - #763
Merged
Merged
Conversation
zhyncs
reviewed
Jul 27, 2024
| - Mistral / Mixtral | ||
| - Gemma / Gemma 2 | ||
| - Qwen / Qwen 2 / Qwen 2 MoE | ||
| - DeepSeek / DeepSeek 2 |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 18, 2026
…ntract behind it was wrong TWO DEFECTS, ONE HIDING THE OTHER. The sgl-project#727 requantized checkpoint (Qwen3.8-27B-INT8-vocabint8-embed) generated token soup under plain TP=3 uneven-DCP: greedy "The capital of France is" returned " a a a a a a a". Swapping ONLY the checkpoint to INT8-yarn1.5, same code path and TP vector, restored coherent output at identical throughput (473 vs 474 tok/s at bs=24), so the defect cost correctness and nothing else. ROOT: A GATE THAT COULD NEVER FIRE. qwen3_5.py selected the embedding method with get_name() == "compressed-tensors", but CompressedTensorsConfig.get_name returns the UNDERSCORE spelling "compressed_tensors" (compressed_tensors.py:262-263). The hyphen is what the checkpoint's quant_method field and the prose use; the config names itself with an underscore. So embedding_quant_config stayed None, the embedding was built DENSE, and the checkpoint's int8 rows were cast into a BF16 embedding with no scale applied while the scale had nowhere to land. Serving said so on every rank: "Parameter model.embed_tokens.weight_scale not found in params_dict" (bench_tp3.log:110-112). Nothing else in the sgl-project#727 lineage was wrong. The ignore-list scan answers True for every embed_tokens prefix on this checkpoint (measured), and get_quant_method dispatches VocabParallelEmbedding correctly. The feature was simply never reached, which is exactly why every test that drove the method directly stayed green. SECOND, MASKED BY THE FIRST. create_weights registered weight and weight_scale without output_dim. VocabParallelEmbedding.weight_loader reads that attribute (vocab_parallel_embedding.py:524, branch at :577) to decide whether to narrow the checkpoint tensor to this rank's rows; without it a parameter takes the "copy onto all gpus" path meant for shard-invariant tensors like gptq's g_idx and asserts at :584. Invisible at tp_size 1 -- the whole vocab IS the local shard, which is why the PP=3 layout never saw it -- and fatal the moment the vocab is row-sharded. Fixing only the gate would have turned token soup into a hard TP boot crash, so both halves land together. The per-row scale shards on dim 0 with the rows it belongs to; slicing one without the other pairs every row with a stranger's scale. THE GATE IS NOW TESTABLE. The predicate moved next to the method it selects as is_compressed_tensors_config, matching the normalized family name so a second spelling cannot re-open this, instead of a bare string duplicated in a model file. TESTS. TestItSurvivesTpVocabSharding loads TP=3 shards through the REAL VocabParallelEmbedding.weight_loader and requires the reduced result to equal the TP=1 reference -- confirmed RED before the fix (AssertionError at vocab_parallel_embedding.py:584). TestTheFamilyGateActuallyMatches binds the predicate to the real config object. The existing suite drove one unsharded partition with a dummy loader (weight_loader=lambda *a, **k: None), which is how both defects passed 17 green tests. Now 18 passed, hermetic on CPU. Selection verified end-to-end at the desk: the fixed gate yields CompressedTensorsEmbeddingMethod with ['weight', 'weight_scale'] both carrying output_dim 0, the old gate yields UnquantizedEmbeddingMethod with ['weight'] alone in bf16 -- the production symptom, reproduced. ruff clean on the files I touched (the unused triton import at qwen3_5.py:24 is pre-existing, verified against HEAD). NOT CLAIMED: that the requantized checkpoint now serves coherently under TP=3 end to end. That needs a boot and has not been run.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 18, 2026
…olve the PP/TP contradiction, and measure step-1 sgl-project#763 BOOT-PROOF. Plain TP=3 with the requantized checkpoint on the fixed tree answers ' Paris.\nThe capital of Germany is Berlin.' where the same checkpoint and the same layout answered ' a a a a a a' before. weight_scale not found in params_dict is now logged 0 times, against 3 (one per rank) pre-fix. The saving landed within 1 MiB of the arithmetic: 404.5 + 402.4 + 404.5 = 1211.4 MiB against a predicted 1212, split evenly because --rank-vocab-ratio is unset. A dense BF16 path cannot produce that reduction, which is what makes the number proof of which method is running -- the code never logs its embedding method, so the absent warning plus the exact byte saving is the evidence available. THE CONTRADICTION IS RESOLVED, AGAINST THE LAYOUT HYPOTHESIS. The open question was why the same checkpoint was reported coherent under PP=3. It was not the layout: boot_735_v7tr_ctg.log (tp_size=1, pp_size=3, vocabint8-embed) carries the same 'weight_scale not found' three times. Under PP the embedding lives on stage 0 alone, and PP0's line is the SCALE only -- PP1/PP2 also report the weight missing simply because they do not own the embedding, a benign pair that appears in the healthy yarn1.5 known-good too, where the scale warning is absent entirely because that checkpoint has no scale tensor. So the embed-owning rank behaved identically under both layouts, and a coherent PP probe on this checkpoint needs an explanation that is not the embedding path. sgl-project#735 STEP-1. The contiguous full plan (SGLANG_PP_LAYER_SET=0-30;31-47;48-63 on --pp-stage-ratio 31,17,16 with FA 7,5,4) booted twice through the ticket's own runner, health 200 at 181 s and 190 s, zero PPLayerSetError/PPMissingLayer, and answered all three seeded probes correctly and identically across both boots. Prefill is +10.1% over the incumbent contiguous baseline: median 5620.2 vs 5106.8 tok/s, within-arm spread under 1%, so roughly fourteen times the noise. TWO MEASUREMENT TRAPS CLEARED FIRST, because the first number was worthless. Repeating a prompt returns in 0.13 s off the radix cache, so every run now carries a UUID salt at the head. Even salted, run 0 is ~4x slower than runs 1-4 (JIT, autotune, allocator growth), so a single-shot measurement compares warmup against steady state -- the one-run-per-arm figure sat inside a 1449-1940 tok/s band on the incumbent alone. prefill_probe.py runs N salted requests and reports the median with the spread beside it. NOT DONE: the ship config was not switched. Step-1 is a perf carrier CANDIDATE on prefill evidence only -- one prompt shape, no decode throughput, no long-run stability. Serving restored to the known-good yarn1.5 unit and verified.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 18, 2026
…ent probe on the int8-vocab checkpoint THE ARTIFACTS DECIDE IT. Every arm booted on vocabint8-embed produced garbage, PP included -- v7pp10, v7pp17, v7pp18, v7pp19, v7pp20, v7tr_gap. Every arm on yarn1.5 answered correctly -- arm1, armA, armA2, armB2, comp4, disc, nohc, and both of today's step1ctg arms. The reported PP coherence belonged to yarn1.5 boots, so the dense-path prediction needs no exception: the defect was never layout-dependent, exactly as the identical 'weight_scale not found' lines under both tp_size=3 and pp_size=3 already indicated. WHY IT LOOKED OTHERWISE, which is the reusable part. The garbage does not appear where a reader looks. CONTENT is the EMPTY STRING and the soup sits in reasoning_content, with FINISH: length because the stream never stops: CONTENT: '' REASONING: ' The wordirikao...EventManager_pars...' The model never emits a closing think marker, so the qwen3 reasoning parser takes the whole degenerate stream as reasoning and leaves content empty. A probe that prints content alone sees '', which reads as "empty answer" rather than "the model is broken". That is the same blindness that let the GATE 0 this ticket had already specified go unrun: a coherence check must read reasoning_content as well, or use the native /generate route, which has no parser in front of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.