float32

[mars1248]

I see that flash attention only supports bfloat16 and fp16. How can solve the problem of exceeding 16 bit accuracy in the calculation process?

[tridao]

You can use standard attention for fp32. We focus on fp16 and bf16 since folks who commonly care about speed already use these formats.