INT8 inference issues w8a8 #4
Description
Hi am am trying to use the Zentorch modules via fp32 with in8 for w8a8, currently i've tried using Quark compile Mistral 7b 0.2 and now also PytorchAO compiled W8a8.
Pytorch W8a8 works as expected without ZenTorch.
Tested from PyTorch ans associated ZenTorch version 2.4 -> 2.8
PyTorch version 2 7 0 (w8a8).txt
See attached for event log and associated basic testing script.
Noting: assuming again this may be due to enforced AVX512 for int8 (which probably should not be in this case), but certainly no mention of it this time.