Repository navigation
[docs] deepgemm - #46361
[docs] deepgemm#46361
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
|
||
| ## UE8M0 scale format | ||
|
|
||
| DeepSeek V4-style checkpoints store FP8 weight scales in the packed `float8_e8m0fnu` format instead of `float32`. Pass `scale_fmt="ue8m0"` to load these checkpoints. UE8M0 scales must take the DeepGEMM path (the Triton fallback only reads `float32` scales) so the same hardware and `kernels` requirements apply. |
There was a problem hiding this comment.
passing scale_fmt="ue8m0" is unnecesary, normally it should be in the quant config shipped with the model, and we try to honor it during weight init. one problem though is that even if it's specified, the on-disk (safetensors) scales need to respect it. this is not the case in https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Base for example where scale_fmt="ue8m0" but the safetensors stores them as fp32 🥲. i forgot to mention this in my previous PR, @ArthurZucker do you think we should cast the scales to ue8m0 if scale_fmt="ue8m0" or keep them in the on-disk format ?
IlyasMoutawwakil
left a comment
IlyasMoutawwakil
left a comment
There was a problem hiding this comment.
thanks a lot @stevhliu ! i left some comments
|
i also updated the docs with #46407 :) |
* docs * sonicmoe * polish * feedback * triton
* docs * sonicmoe * polish * feedback * triton
* docs * sonicmoe * polish * feedback * triton
adds docs for #45634:
deepgemmwith fp8/fp4 quantized experts,deepgemm_megamoeandsonicmoesections