fix(qwen3): load lm_head unquantized for GPTQ/AWQ checkpoints - #2348
Open
sjmignot wants to merge 2 commits into
Open
fix(qwen3): load lm_head unquantized for GPTQ/AWQ checkpoints#2348sjmignot wants to merge 2 commits into
sjmignot wants to merge 2 commits into
Conversation
Standard GPTQ/AWQ Qwen3 checkpoints (e.g. Qwen/Qwen3-8B-AWQ) quantize only the transformer blocks and keep lm_head in full precision (plain lm_head.weight, no qweight/qzeros/scales). qwen3.rs passed the model's quantization_config when building lm_head via ReplicatedLayer::new, so the loader demanded int4 lm_head tensors and crashed with: 'Missing required tensor(s) for gptq_awq_linear at prefix lm_head' Load lm_head unquantized instead, matching gemma.rs/starcoder2.rs. Branched from upstream v0.9.0 (chalk/0.9-fixes).
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 23 4454 3116 790 548 CSS 3 282 252 6 24 CUDA 119 23681 19217 1704 2760 Dockerfile 1 35 19 9 7 HTML 2 27 27 0 0 JavaScript 3 577 562 12 3 Jinja2 7 694 656 5 33 JSON 27 15864 15861 0 3 Makefile 1 18 16 0 2 MDX 36 5901 0 4327 1574 Metal Shading Lan| 37 14287 11284 1136 1867 PowerShell 1 657 571 31 55 Python 151 12284 10191 484 1609 Shell 3 1068 849 116 103 Plain Text 53 10687 0 9209 1478 TOML 28 1368 1189 39 140 TypeScript 11 1649 1410 66 173 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 3 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 273 11726 0 8736 2990 |- BASH 23 295 217 46 32 |- Dockerfile 2 5 5 0 0 |- JSON 6 289 289 0 0 |- PowerShell 1 1 1 0 0 |- Python 135 7239 6021 310 908 |- Rust 62 3820 2838 388 594 |- TOML 7 77 65 0 12 (Total) 23452 9436 9480 4536 ───────────────────────────────────────────────────────────────────────────────── Rust 674 300652 267964 5856 26832 |- Markdown 413 9761 452 8126 1183 (Total) 310413 268416 13982 28015 ───────────────────────────────────────────────────────────────────────────────── Svelte 19 1969 1826 51 92 |- CSS 1 4 4 0 0 |- JavaScript 19 921 767 25 129 (Total) 2894 2597 76 221 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1478 430621 345918 41520 43183 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
The marlin locks buffer was allocated as `out_dim / pack_factor`, which only matches the kernel's requirement for 8-bit (pack_factor 4 -> out_dim/4). For 4-bit (pack_factor 8 -> out_dim/8) it under-allocates by 2x, so marlin writes its cross-block reduction locks out of bounds. Depending on GPU memory layout this is silent corruption (garbage output) or CUDA_ERROR_ILLEGAL_ADDRESS (observed with AWQ int4 on A100/sm_80; masked on L4/sm_89). Size it as (out_dim / min_thread_n) * max_par = out_dim/4 for all bit widths, matching vLLM's MarlinWorkspace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standard GPTQ/AWQ Qwen3 checkpoints (e.g. Qwen/Qwen3-8B-AWQ) quantize only the transformer blocks and keep lm_head in full precision (plain lm_head.weight, no qweight/qzeros/scales). qwen3.rs passed the model's quantization_config when building lm_head via ReplicatedLayer::new, so the loader demanded int4 lm_head tensors and crashed with:
'Missing required tensor(s) for gptq_awq_linear at prefix lm_head'
Load lm_head unquantized instead, matching gemma.rs/starcoder2.rs.
Branched from upstream v0.9.0 (chalk/0.9-fixes).