Skip to content

Support new format AWQ MoE models - #2358

Open
guoqingbao wants to merge 2 commits into
EricLBuehler:masterfrom
guoqingbao:master
Open

Support new format AWQ MoE models#2358
guoqingbao wants to merge 2 commits into
EricLBuehler:masterfrom
guoqingbao:master

Conversation

@guoqingbao

Copy link
Copy Markdown
Contributor

Add support for AWQ MoE model support, tested with:

cargo build --release --features cuda,nccl
CUDA_VISIBLE_DEVICES=0 target/release/mistralrs run -m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

Perf w/o flashattn

CLI time to first token: 0.14s
Prompt: 579 tokens, 4226.28 T/s
Decode: 1550 tokens, 61.83 T/s
Prefix cache: 1 hits / 2 turns
Sampling: temp=1, top_k=40, top_p=0.95, min_p=off

@github-actions

Copy link
Copy Markdown
Code Metrics Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 23         4454         3116          790          548
 CSS                       3          282          252            6           24
 CUDA                    121        25439        20835         1707         2897
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          578          563           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        17989        17986            0            3
 Makefile                  1           18           16            0            2
 MDX                      36         6285            0         4616         1669
 Metal Shading Lan|       37        14409        11401         1136         1872
 PowerShell                1          657          571           31           55
 Python                  151        12539        10418          480         1641
 Shell                     3         1071          852          115          104
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1389         1209           39          141
 TypeScript               11         1649         1410           66          173
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                273        12055            0         8981         3074
 |- BASH                  24          300          221           47           32
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6120          306          923
 |- Rust                  62         3840         2849          394          597
 |- TOML                   7           92           78            0           14
 (Total)                            23940         9570         9728         4642
─────────────────────────────────────────────────────────────────────────────────
 Rust                    701       331485       296772         5817        28896
 |- Markdown             418         9947          452         8304         1191
 (Total)                           341432       297224        14121        30087
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1507       466797       378971        42194        45632
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant