Skip to content

[1/3] fix dsv3 awq issue - #4556

Merged
zhyncs merged 13 commits into
sgl-project:mainfrom
AniZpZ:awq-fix
Mar 22, 2025
Merged

zhyncs merged 13 commits into
sgl-project:mainfrom
AniZpZ:awq-fix

Conversation

@AniZpZ

@AniZpZ AniZpZ commented Mar 18, 2025

Copy link
Copy Markdown
Collaborator

Motivation

related issue: #4462

Currently dsv3 awq suffers accuracy problem that may due to the overflow of fp16.

Modifications

co-author: @Hongbosherlock

We add support int4 -> bf16 in awq_dequantize to inpelement int4-bf16 quantization.

Checklist

@Hongbosherlock

Hongbosherlock commented Mar 18, 2025 •

Copy link
Copy Markdown
Contributor

Both float16 and bfloat16 tests have passed on A800
image

@AniZpZ AniZpZ changed the title [1/2] fix dsv3 awq issue [1/3] fix dsv3 awq issue Mar 18, 2025
@Hongbosherlock

Hongbosherlock commented Mar 18, 2025 •

Copy link
Copy Markdown
Contributor

@zhyncs This kernel supports GPUs with __CUDA_ARCH__ >= 800, including A100 and A800, which are commonly used to run AWQ models.

@laixinn

laixinn commented Mar 19, 2025 •

Copy link
Copy Markdown
Contributor

FMA operations in dequantize_s4_to_bf16x2 would be wired when we shift 4, 8, 12 respectively.
A simple subtraction can do this conversion. Here is the benchmarking between these two implementations and the FP16.

Shape __hsub2 (us) __hfma2 (us) fp16 (us)
(3584, 448) 7.10487 7.29561 7.20024
(3584, 576) 6.96182 7.10487 7.29561
(3584, 4736) 7.10487 7.24792 7.34329
(3584, 16) 10.20432 7.05719 7.34329
(3584, 32) 7.05719 11.44409 6.91414
(3584, 64) 6.96182 7.20024 7.05719
(3584, 128) 8.05855 6.91414 6.72340
(18944, 448) 6.48499 6.62804 6.48499
(18944, 576) 6.77109 6.91414 7.20024
(18944, 4736) 7.10487 7.00951 6.96182
(18944, 16) 8.15392 6.43730 6.67572
(18944, 32) 7.00951 7.39098 6.91414
(18944, 64) 10.68115 6.81877 6.96182
(18944, 128) 10.96725 7.20024 7.15256
(128, 448) 7.10487 6.96182 6.77109
(128, 576) 6.81877 6.91414 6.77109
(128, 4736) 8.15392 7.00951 6.86646
(128, 16) 6.58035 6.77109 7.77245
(128, 32) 6.62804 8.44002 6.62804
(128, 64) 6.67572 6.77109 6.77109
(128, 128) 6.67572 6.86646 6.67572
(256, 448) 6.86646 6.96182 7.05719
(256, 576) 6.86646 7.05719 7.24792
(256, 4736) 6.96182 7.15256 7.15256
(256, 16) 6.67572 6.96182 10.91957
(256, 32) 6.77109 6.81877 7.10487
(256, 64) 6.72340 7.10487 6.86646
(256, 128) 10.44273 7.05719 6.62804
(512, 448) 6.86646 9.10759 7.00951
(512, 576) 6.67572 8.34465 6.77109
(512, 4736) 7.10487 7.00951 7.24792
(512, 16) 6.77109 8.15392 6.91414
(512, 32) 7.00951 6.96182 6.77109
(512, 64) 6.86646 7.10487 6.91414
(512, 128) 6.96182 7.29561 7.00951
(1024, 448) 6.91414 7.24792 7.29561
(1024, 576) 6.96182 7.20024 6.96182
(1024, 4736) 7.20024 7.05719 7.20024
(1024, 16) 7.77245 6.67572 8.48770
(1024, 32) 6.91414 6.86646 6.77109
(1024, 64) 7.15256 7.24792 6.86646
(1024, 128) 7.00951 6.81877 7.00951
avg 7.35124 7.25247 7.11169

@AniZpZ

AniZpZ commented Mar 19, 2025

Copy link
Copy Markdown
Collaborator Author

FMA operations in dequantize_s4_to_bf16x2 would be wired when we shift 4, 8, 12 respectively. A simple subtraction can do this conversion. Here is the benchmarking between these two implementations.

Parameters __hmsub2 (us) __hfma2 (us)
(3584, 448) 7.10487 7.29561
(3584, 576) 6.96182 7.10487
(3584, 4736) 7.10487 7.24792
(3584, 16) 10.20432 7.05719
(3584, 32) 7.05719 11.44409
(3584, 64) 6.96182 7.20024
(3584, 128) 8.05855 6.91414
(18944, 448) 6.48499 6.62804
(18944, 576) 6.77109 6.91414
(18944, 4736) 7.10487 7.00951
(18944, 16) 8.15392 6.43730
(18944, 32) 7.00951 7.39098
(18944, 64) 10.68115 6.81877
(18944, 128) 10.96725 7.20024
(128, 448) 7.10487 6.96182
(128, 576) 6.81877 6.91414
(128, 4736) 8.15392 7.00951
(128, 16) 6.58035 6.77109
(128, 32) 6.62804 8.44002
(128, 64) 6.67572 6.77109
(128, 128) 6.67572 6.86646
(256, 448) 6.86646 6.96182
(256, 576) 6.86646 7.05719
(256, 4736) 6.96182 7.15256
(256, 16) 6.67572 6.96182
(256, 32) 6.77109 6.81877
(256, 64) 6.72340 7.10487
(256, 128) 10.44273 7.05719
(512, 448) 6.86646 9.10759
(512, 576) 6.67572 8.34465
(512, 4736) 7.10487 7.00951
(512, 16) 6.77109 8.15392
(512, 32) 7.00951 6.96182
(512, 64) 6.86646 7.10487
(512, 128) 6.96182 7.29561
(1024, 448) 6.91414 7.24792
(1024, 576) 6.96182 7.20024
(1024, 4736) 7.20024 7.05719
(1024, 16) 7.77245 6.67572
(1024, 32) 6.91414 6.86646
(1024, 64) 7.15256 7.24792
(1024, 128) 7.00951 6.81877
Avg 7.35124 7.25247

Thx for detailed benchmark data!
This PR is ready for review now @zhyncs @ispobock

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants