Qualcomm AI Engine Direct - Enable 4 bits BW quantization

[chunit-quic]

• Add QNN_QUANTIZATION_ENCODING_BW... confings for qnn wrapper
• Add 4 bits quant config
• Add 4 bits quant single op tests
• Add per channel weight setting for quantizer
• Fix convert_to_linear error
• Refine quantizer

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/2506

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2c0be5e with merge base 588c391 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[cccclai]

Looks like the quantization is applied to the whole model, is it possible to have more granularity like specifying the quantization type for each op

cc: @jerryzh168

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai merged this pull request in 3270d22.

[chunit-quic]

Hi Chen,

is it possible to have more granularity like specifying the quantization type for each op

Our quantizer provides two ways to quantize op with different configs. One thing to note is that the mixed precision(WIP) is not completed yet, and it might fail to validate some op.

1. Use add_custom_quant_annotations function
   This function requires user to provide a sequence of annotation functions. It has the biggest flexibility to configure ops annotation setting
2. 8bits and 16 bits case
   In most cases we quantize op in 8a8w (activation/weight) and 16a16w format. (4 bits quantization only works on the weight of few ops. You may consider it as a special setting). Using add_16bit_quant_ops precisely for those ops require 16 bits act, and the rest left in 8 bits act is another choice .

Finally, we could consider to refine _get_quant_config a bits, and allow users to declare their own {op :config} dict. This way needs us to submit another PR.

[cccclai]

Thank you! On Monday's meeting, we brought up the json configurable quantization, is it something we can target for?

[chunit-quic]

Thank you! On Monday's meeting, we brought up the json configurable quantization, is it something we can target for?

In case I have any misuderstanding. May you kindly give us an example to show what would the content of json be like? Thank you. :)

[chunit-quic]

Thank you! On Monday's meeting, we brought up the json configurable quantization, is it something we can target for?

Hi @cccclai,

Just a gentle ping. Would you mind to give us an example of the content of json? So that we could think about how to fullfill your requirements. Thank you. :D