Background Description
Some models can produce activations that are beyond the range of fp16. In that scenario, usage of Q8_1 to quantize the activations can be futile, see discussion in #194.
Hence, it would be prudent to change all quantization types using Q8_1 for matrix multiplications to use something else.
Alternatively, one may replace the fp16 block scale and block sum in Q8_1 with bf16.
Possible Refactor Approaches
No response
Background Description
Some models can produce activations that are beyond the range of
fp16. In that scenario, usage ofQ8_1to quantize the activations can be futile, see discussion in #194.Hence, it would be prudent to change all quantization types using
Q8_1for matrix multiplications to use something else.Alternatively, one may replace the
fp16block scale and block sum inQ8_1withbf16.Possible Refactor Approaches
No response