Use Core ML Quantizer in Llama Export

[YifanShenSZ]

This PR is an initial step to add Core ML quantizer in Llama export. We start with "quantize model with XNNPack quantizer then fully delegate to Core ML backend". "Quantize with Core ML quantizer" is under development

This PR does 2 things:

1. Add Core ML quantizer options then use them in Llama export
2. Use different iOS versions for different features: fp16 model can run on iOS 15, while 8a8w quantization requires iOS 17, and 4w quantization requires iOS 18

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4458

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 68d345c with merge base 5a20a49 ():

NEW FAILURE - The following job has failed:

• pull / test-export-llava-linux / linux-job (gh)
  RuntimeError: Command docker exec -t e81de0580fa36108059fa62166e293629e6eff10d8e7ff27ff52c4798276d2da /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[YifanShenSZ]

Add @cccclai @shoumikhin as reviewers

(smh I cannot assign reviewer 😂 so @ you here)

[YifanShenSZ]

@cymbalrush since now we have skip_model_load=True in preprocess, the compute unit in llama export script is no longer necessary, right?

-        # using `ComputeUnit.ALL` can increase the model load time, default to `ComputeUnit.CPU_AND_GPU`
-        compute_unit=ct.ComputeUnit[ct.ComputeUnit.CPU_AND_GPU.name.upper()],

[cccclai]

Thanks for putting up the PR! For the change in llama_transformer.py, actually there was some CI regression as shown in #3786.

[YifanShenSZ]

Thanks for putting up the PR! For the change in llama_transformer.py, actually there was some CI regression as shown in #3786.

Yeah this is quite interesting

1. I can export stories 110M on my local without issue
2. I took a look at the failed CI in #3786, and that is the same error we discussed on slack when I tried to tag mutated buffer... So it's probably a flakiness e.g. in exported program constructor?

Anyway, since that's just a minor fix, I reverted the llama_transformer.py and sdpa.py changes to make CI green.

[cccclai]

Looks good in general. Can we fix the CI and rename the coreml_xnnpack, coreml_xnnpack_qc4?

[cymbalrush]

@cymbalrush since now we have skip_model_load=True in preprocess, the compute unit in llama export script is no longer necessary, right?

-        # using `ComputeUnit.ALL` can increase the model load time, default to `ComputeUnit.CPU_AND_GPU`
-        compute_unit=ct.ComputeUnit[ct.ComputeUnit.CPU_AND_GPU.name.upper()],

We still want it CPU_AND_GPU , I am concerned about the model load time.

[YifanShenSZ]

We still want it CPU_AND_GPU , I am concerned about the model load time.

Ok, reverted that change to still keep CPU_AND_GPU there

[cccclai]

There is also a lint error in the CI. Mind addressing it?

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[YifanShenSZ]

There is also a lint error in the CI. Mind addressing it?

Applied the lint fix

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

there are some pyre error but they aren't part of the oss ci...

[cccclai]

lint error again 😅

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[YifanShenSZ]

lint error again 😅

Fixed 😅

[facebook-github-bot]

@cccclai merged this pull request in 6bfefa8.