Skip to content

Update supported models - #763

Merged
zhyncs merged 1 commit into
mainfrom
update-support-models
Jul 27, 2024
Merged

zhyncs merged 1 commit into
mainfrom
update-support-models

Conversation

@hnyls2002

Copy link
Copy Markdown
Collaborator

No description provided.

Comment thread README.md
- Mistral / Mixtral
- Gemma / Gemma 2
- Qwen / Qwen 2 / Qwen 2 MoE
- DeepSeek / DeepSeek 2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DeepSeek MOE?

@zhyncs
zhyncs merged commit ba29504 into main Jul 27, 2024
@zhyncs
zhyncs deleted the update-support-models branch July 27, 2024 05:53
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 18, 2026
…ntract behind it was wrong

TWO DEFECTS, ONE HIDING THE OTHER. The sgl-project#727 requantized checkpoint
(Qwen3.8-27B-INT8-vocabint8-embed) generated token soup under plain TP=3
uneven-DCP: greedy "The capital of France is" returned " a a a a a a a". Swapping
ONLY the checkpoint to INT8-yarn1.5, same code path and TP vector, restored
coherent output at identical throughput (473 vs 474 tok/s at bs=24), so the
defect cost correctness and nothing else.

ROOT: A GATE THAT COULD NEVER FIRE. qwen3_5.py selected the embedding method with
get_name() == "compressed-tensors", but CompressedTensorsConfig.get_name returns
the UNDERSCORE spelling "compressed_tensors" (compressed_tensors.py:262-263). The
hyphen is what the checkpoint's quant_method field and the prose use; the config
names itself with an underscore. So embedding_quant_config stayed None, the
embedding was built DENSE, and the checkpoint's int8 rows were cast into a BF16
embedding with no scale applied while the scale had nowhere to land. Serving said
so on every rank: "Parameter model.embed_tokens.weight_scale not found in
params_dict" (bench_tp3.log:110-112).

Nothing else in the sgl-project#727 lineage was wrong. The ignore-list scan answers True for
every embed_tokens prefix on this checkpoint (measured), and get_quant_method
dispatches VocabParallelEmbedding correctly. The feature was simply never reached,
which is exactly why every test that drove the method directly stayed green.

SECOND, MASKED BY THE FIRST. create_weights registered weight and weight_scale
without output_dim. VocabParallelEmbedding.weight_loader reads that attribute
(vocab_parallel_embedding.py:524, branch at :577) to decide whether to narrow the
checkpoint tensor to this rank's rows; without it a parameter takes the "copy onto
all gpus" path meant for shard-invariant tensors like gptq's g_idx and asserts at
:584. Invisible at tp_size 1 -- the whole vocab IS the local shard, which is why
the PP=3 layout never saw it -- and fatal the moment the vocab is row-sharded.
Fixing only the gate would have turned token soup into a hard TP boot crash, so
both halves land together. The per-row scale shards on dim 0 with the rows it
belongs to; slicing one without the other pairs every row with a stranger's scale.

THE GATE IS NOW TESTABLE. The predicate moved next to the method it selects as
is_compressed_tensors_config, matching the normalized family name so a second
spelling cannot re-open this, instead of a bare string duplicated in a model file.

TESTS. TestItSurvivesTpVocabSharding loads TP=3 shards through the REAL
VocabParallelEmbedding.weight_loader and requires the reduced result to equal the
TP=1 reference -- confirmed RED before the fix (AssertionError at
vocab_parallel_embedding.py:584). TestTheFamilyGateActuallyMatches binds the
predicate to the real config object. The existing suite drove one unsharded
partition with a dummy loader (weight_loader=lambda *a, **k: None), which is how
both defects passed 17 green tests. Now 18 passed, hermetic on CPU. Selection
verified end-to-end at the desk: the fixed gate yields
CompressedTensorsEmbeddingMethod with ['weight', 'weight_scale'] both carrying
output_dim 0, the old gate yields UnquantizedEmbeddingMethod with ['weight'] alone
in bf16 -- the production symptom, reproduced. ruff clean on the files I touched
(the unused triton import at qwen3_5.py:24 is pre-existing, verified against HEAD).

NOT CLAIMED: that the requantized checkpoint now serves coherently under TP=3
end to end. That needs a boot and has not been run.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 18, 2026
…olve the PP/TP contradiction, and measure step-1

sgl-project#763 BOOT-PROOF. Plain TP=3 with the requantized checkpoint on the fixed tree
answers ' Paris.\nThe capital of Germany is Berlin.' where the same checkpoint
and the same layout answered ' a a a a a a' before. weight_scale not found in
params_dict is now logged 0 times, against 3 (one per rank) pre-fix. The saving
landed within 1 MiB of the arithmetic: 404.5 + 402.4 + 404.5 = 1211.4 MiB
against a predicted 1212, split evenly because --rank-vocab-ratio is unset. A
dense BF16 path cannot produce that reduction, which is what makes the number
proof of which method is running -- the code never logs its embedding method,
so the absent warning plus the exact byte saving is the evidence available.

THE CONTRADICTION IS RESOLVED, AGAINST THE LAYOUT HYPOTHESIS. The open question
was why the same checkpoint was reported coherent under PP=3. It was not the
layout: boot_735_v7tr_ctg.log (tp_size=1, pp_size=3, vocabint8-embed) carries
the same 'weight_scale not found' three times. Under PP the embedding lives on
stage 0 alone, and PP0's line is the SCALE only -- PP1/PP2 also report the
weight missing simply because they do not own the embedding, a benign pair that
appears in the healthy yarn1.5 known-good too, where the scale warning is
absent entirely because that checkpoint has no scale tensor. So the
embed-owning rank behaved identically under both layouts, and a coherent PP
probe on this checkpoint needs an explanation that is not the embedding path.

sgl-project#735 STEP-1. The contiguous full plan (SGLANG_PP_LAYER_SET=0-30;31-47;48-63 on
--pp-stage-ratio 31,17,16 with FA 7,5,4) booted twice through the ticket's own
runner, health 200 at 181 s and 190 s, zero PPLayerSetError/PPMissingLayer, and
answered all three seeded probes correctly and identically across both boots.
Prefill is +10.1% over the incumbent contiguous baseline: median 5620.2 vs
5106.8 tok/s, within-arm spread under 1%, so roughly fourteen times the noise.

TWO MEASUREMENT TRAPS CLEARED FIRST, because the first number was worthless.
Repeating a prompt returns in 0.13 s off the radix cache, so every run now
carries a UUID salt at the head. Even salted, run 0 is ~4x slower than runs 1-4
(JIT, autotune, allocator growth), so a single-shot measurement compares warmup
against steady state -- the one-run-per-arm figure sat inside a 1449-1940 tok/s
band on the incumbent alone. prefill_probe.py runs N salted requests and reports
the median with the spread beside it.

NOT DONE: the ship config was not switched. Step-1 is a perf carrier CANDIDATE
on prefill evidence only -- one prompt shape, no decode throughput, no long-run
stability. Serving restored to the known-good yarn1.5 unit and verified.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 18, 2026
…ent probe on the int8-vocab checkpoint

THE ARTIFACTS DECIDE IT. Every arm booted on vocabint8-embed produced garbage,
PP included -- v7pp10, v7pp17, v7pp18, v7pp19, v7pp20, v7tr_gap. Every arm on
yarn1.5 answered correctly -- arm1, armA, armA2, armB2, comp4, disc, nohc, and
both of today's step1ctg arms. The reported PP coherence belonged to yarn1.5
boots, so the dense-path prediction needs no exception: the defect was never
layout-dependent, exactly as the identical 'weight_scale not found' lines under
both tp_size=3 and pp_size=3 already indicated.

WHY IT LOOKED OTHERWISE, which is the reusable part. The garbage does not
appear where a reader looks. CONTENT is the EMPTY STRING and the soup sits in
reasoning_content, with FINISH: length because the stream never stops:

  CONTENT: ''
  REASONING: ' The wordirikao...EventManager_pars...'

The model never emits a closing think marker, so the qwen3 reasoning parser
takes the whole degenerate stream as reasoning and leaves content empty. A probe
that prints content alone sees '', which reads as "empty answer" rather than
"the model is broken". That is the same blindness that let the GATE 0 this
ticket had already specified go unrun: a coherence check must read
reasoning_content as well, or use the native /generate route, which has no
parser in front of it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants