Skip to content

Commit c004efe

Browse files
yifan_shen3facebook-github-bot
yifan_shen3
authored and
facebook-github-bot
committed
Update Core ML Backend Doc (#3188)
Summary: Update Core ML backend doc on: 1. Partitioner 2. Quantizer Pull Request resolved: #3188 Reviewed By: shoumikhin Differential Revision: D56481126 Pulled By: cccclai fbshipit-source-id: 925a107a210094e035a816a15c70d9aedd5bd369
1 parent ee8c3a6 commit c004efe

File tree

4 files changed

+162
-58
lines changed

4 files changed

+162
-58
lines changed

‎backends/apple/coreml/README.md

Lines changed: 90 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -6,58 +6,123 @@ Core ML is an optimized framework for running machine learning models on Apple d
66

77
## Layout
88
- `compiler/` : Lowers a module to Core ML backend.
9+
- `partition/`: Partitions a module fully or partially to Core ML backend.
10+
- `quantizer/`: Quantizes a module in Core ML favored scheme.
911
- `scripts/` : Scripts for installing dependencies and running tests.
1012
- `runtime/`: Core ML delegate runtime implementation.
1113
- `inmemoryfs`: InMemory filesystem implementation used to serialize/de-serialize AOT blob.
1214
- `kvstore`: Persistent Key-Value store implementation.
1315
- `delegate`: Runtime implementation.
1416
- `include` : Public headers.
15-
- `tests` : Tests for Core ML delegate.
16-
- `workspace` : Xcode workspace for tests.
17+
- `sdk` : SDK implementation.
18+
- `tests` : Unit tests.
19+
- `workspace` : Xcode workspace for the runtime.
1720
- `third-party/`: External dependencies.
1821

19-
## Help & Improvements
20-
If you have problems or questions or have suggestions for ways to make
21-
implementation and testing better, please create an issue on [github](https://www.github.com/pytorch/executorch/issues).
22+
## Partition and Delegation
2223

23-
## Delegation
24-
25-
For delegating the Program to the **Core ML** backend, the client must be responsible for calling `to_backend` with the **CoreMLBackend** tag.
24+
To delegate a Program to the **Core ML** backend, the client must call `to_backend` with the **CoreMLPartitioner**.
2625

2726
```python
28-
import executorch.exir as exir
2927
import torch
30-
31-
from torch.export import export
32-
33-
from executorch.exir import to_edge
34-
35-
from executorch.exir.backend.backend_api import to_backend
28+
import executorch.exir
3629

3730
from executorch.backends.apple.coreml.compiler import CoreMLBackend
31+
from executorch.backends.apple.coreml.partition.coreml_partitioner import CoreMLPartitioner
3832

39-
class LowerableSubModel(torch.nn.Module):
33+
class Model(torch.nn.Module):
4034
def __init__(self):
4135
super().__init__()
4236

4337
def forward(self, x):
4438
return torch.sin(x)
4539

46-
# Convert the lowerable module to Edge IR Representation
47-
to_be_lowered = LowerableSubModel()
48-
example_input = (torch.ones(1), )
49-
to_be_lowered_exir_submodule = to_edge(export(to_be_lowered, example_input))
40+
source_model = Model()
41+
example_inputs = (torch.ones(1), )
42+
43+
# Export the source model to Edge IR representation
44+
aten_program = torch.export.export(source_model, example_inputs)
45+
edge_program_manager = executorch.exir.to_edge(aten_program)
46+
47+
# Delegate to Core ML backend
48+
delegated_program_manager = edge_program_manager.to_backend(CoreMLPartitioner())
5049

51-
# Lower to Core ML backend
52-
lowered_module = to_backend('CoreMLBackend', to_be_lowered_exir_submodule.exported_program, [])
50+
# Serialize delegated program
51+
executorch_program = delegated_program_manager.to_executorch()
52+
with open("model.pte", "wb") as f:
53+
f.write(executorch_program.buffer)
5354
```
5455

55-
Currently, the **Core ML** backend delegates the whole module to **Core ML**. If a specific op is not supported by the **Core ML** backend then the `to_backend` call would throw an exception. We will be adding a **Core ML Partitioner** to resolve the issue.
56+
The module will be fully or partially delegated to **Core ML**, depending on whether all or part of ops are supported by the **Core ML** backend. User may force skip certain ops by `CoreMLPartitioner(skip_ops_for_coreml_delegation=...)`
57+
58+
The `to_backend` implementation is a thin wrapper over [coremltools](https://apple.github.io/coremltools/docs-guides/), `coremltools` is responsible for converting an **ExportedProgram** to a **MLModel**. The converted **MLModel** data is saved, flattened, and returned as bytes to **ExecuTorch**.
59+
60+
## Quantization
5661

57-
The `to_backend` implementation is a thin wrapper over `coremltools`, `coremltools` is responsible for converting an **ExportedProgram** to a **MLModel**. The converted **MLModel** data is saved, flattened, and returned as bytes to **ExecuTorch**.
62+
To quantize a Program in a Core ML favored way, the client may utilize **CoreMLQuantizer**.
63+
64+
```python
65+
import torch
66+
import executorch.exir
67+
68+
from torch._export import capture_pre_autograd_graph
69+
from torch.ao.quantization.quantize_pt2e import (
70+
convert_pt2e,
71+
prepare_pt2e,
72+
prepare_qat_pt2e,
73+
)
74+
75+
from executorch.backends.apple.coreml.quantizer.coreml_quantizer import CoreMLQuantizer
76+
from coremltools.optimize.torch.quantization.quantization_config import (
77+
LinearQuantizerConfig,
78+
QuantizationScheme,
79+
)
80+
81+
class Model(torch.nn.Module):
82+
def __init__(self) -> None:
83+
super().__init__()
84+
self.conv = torch.nn.Conv2d(
85+
in_channels=3, out_channels=16, kernel_size=3, padding=1
86+
)
87+
self.relu = torch.nn.ReLU()
88+
89+
def forward(self, x: torch.Tensor) -> torch.Tensor:
90+
a = self.conv(x)
91+
return self.relu(a)
92+
93+
source_model = Model()
94+
example_inputs = (torch.randn((1, 3, 256, 256)), )
95+
96+
pre_autograd_aten_dialect = capture_pre_autograd_graph(model, example_inputs)
97+
98+
quantization_config = LinearQuantizerConfig.from_dict(
99+
{
100+
"global_config": {
101+
"quantization_scheme": QuantizationScheme.symmetric,
102+
"activation_dtype": torch.uint8,
103+
"weight_dtype": torch.int8,
104+
"weight_per_channel": True,
105+
}
106+
}
107+
)
108+
quantizer = CoreMLQuantizer(quantization_config)
109+
110+
# For post-training quantization, use `prepare_pt2e`
111+
# For quantization-aware trainin,g use `prepare_qat_pt2e`
112+
prepared_graph = prepare_pt2e(pre_autograd_aten_dialect, quantizer)
113+
114+
prepared_graph(*example_inputs)
115+
converted_graph = convert_pt2e(prepared_graph)
116+
```
117+
118+
The `converted_graph` is the quantized torch model, and can be delegated to **Core ML** similarly through **CoreMLPartitioner**
58119

59120
## Runtime
60121

61-
To execute a **Core ML** delegated **Program**, the client must link to the `coremldelegate` library. Once linked there are no additional steps required, **ExecuTorch** when running the **Program** would call the **Core ML** runtime to execute the **Core ML** delegated part of the **Program**.
122+
To execute a Core ML delegated program, the application must link to the `coremldelegate` library. Once linked there are no additional steps required, ExecuTorch when running the program would call the Core ML runtime to execute the Core ML delegated part of the program.
62123

63124
Please follow the instructions described in the [Core ML setup](/backends/apple/coreml/setup.md) to link the `coremldelegate` library.
125+
126+
## Help & Improvements
127+
If you have problems or questions or have suggestions for ways to make
128+
implementation and testing better, please create an issue on [github](https://www.github.com/pytorch/executorch/issues).

‎backends/apple/coreml/setup.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ python3 -m examples.apple.coreml.scripts.export --model_name add
2929
4. You can now integrate the **Core ML** backend in code.
3030

3131
```python
32-
# Lower to Core ML backend
33-
lowered_module = to_backend('CoreMLBackend', to_be_lowered_exir_submodule, [])
32+
# Delegate to Core ML backend
33+
delegated_program_manager = edge_program_manager.to_backend(CoreMLPartitioner())
3434
```
3535

3636

@@ -46,15 +46,15 @@ lowered_module = to_backend('CoreMLBackend', to_be_lowered_exir_submodule, [])
4646
xcode-select --install
4747
```
4848

49-
2. Build **Core ML** delegate. The following will create a `executorch.xcframework` in `cmake-out` directory.
49+
4. Build **Core ML** delegate. The following will create `executorch.xcframework` and `coreml_backend.xcframework` in the `cmake-out` directory.
5050

5151
```bash
5252
cd executorch
5353
./build/build_apple_frameworks.sh --Release --coreml
5454
```
55-
3. Open the project in Xcode, and drag the `executorch.xcframework` generated from Step 2 to Frameworks.
55+
5. Open the project in Xcode, and drag `executorch.xcframework` and `coreml_backend.xcframework` frameworks generated from Step 2 to Frameworks.
5656

57-
4. Go to project Target’s Build Phases - Link Binaries With Libraries, click the + sign, and add the following frameworks:
57+
6. Go to project Target’s Build Phases - Link Binaries With Libraries, click the + sign, and add the following frameworks:
5858

5959
```
6060
executorch.xcframework
@@ -63,9 +63,9 @@ coreml_backend.xcframework
6363

6464
5. Go to project Target’s Build Phases - Link Binaries With Libraries, click the + sign, and add the following frameworks.
6565
```
66-
- Accelerate.framework
67-
- CoreML.framework
68-
- libsqlite3.tbd
66+
Accelerate.framework
67+
CoreML.framework
68+
libsqlite3.tbd
6969
```
7070

7171
6. The target could now run a **Core ML** delegated **Program**.

‎docs/source/build-run-coreml.md

Lines changed: 46 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Building and Running ExecuTorch with Core ML Backend
22

3-
Core ML delegate uses Core ML apis to enable running neural networks via Apple's hardware acceleration. For more about coreml you can read [here](https://developer.apple.com/documentation/coreml). In this tutorial we will walk through steps of lowering a PyTorch model to Core ML delegate
3+
Core ML delegate uses Core ML APIs to enable running neural networks via Apple's hardware acceleration. For more about coreml you can read [here](https://developer.apple.com/documentation/coreml). In this tutorial, we will walk through the steps of lowering a PyTorch model to Core ML delegate
44

55

66
::::{grid} 2
@@ -67,22 +67,53 @@ python3 -m examples.apple.coreml.scripts.export --model_name mv3
6767

6868
### Runtime:
6969

70-
**Running the Core ML delegated Program**:
70+
**Running a Core ML delegated Program**:
7171
1. Build the runner.
7272
```bash
7373
cd executorch
7474

75-
# Generates ./coreml_executor_runner.
75+
# Builds `coreml_executor_runner`.
7676
./examples/apple/coreml/scripts/build_executor_runner.sh
7777
```
78-
2. Run the exported program.
78+
2. Run the CoreML delegated program.
7979
```bash
8080
cd executorch
8181

82-
# Runs the exported mv3 model on the Core ML backend.
82+
# Runs the exported mv3 model using the Core ML backend.
8383
./coreml_executor_runner --model_path mv3_coreml_all.pte
8484
```
8585

86+
**Profiling a Core ML delegated Program**:
87+
88+
Note that profiling is supported on [macOS](https://developer.apple.com/macos) >= 14.4.
89+
90+
1. [Optional] Generate an [ETRecord](./sdk-etrecord.rst) when exporting your model.
91+
```bash
92+
cd executorch
93+
94+
# Generates `mv3_coreml_all.pte` and `mv3_coreml_etrecord.bin` files.
95+
python3 -m examples.apple.coreml.scripts.export --model_name mv3 --generate_etrecord
96+
```
97+
98+
2. Build the runner.
99+
```bash
100+
# Builds `coreml_executor_runner`.
101+
./examples/apple/coreml/scripts/build_executor_runner.sh
102+
```
103+
3. Run and generate an [ETDump](./sdk-etdump.md).
104+
```bash
105+
cd executorch
106+
107+
# Generate the ETDump file.
108+
./coreml_executor_runner --model_path mv3_coreml_all.pte --profile_model --etdump_path etdump.etdp
109+
```
110+
111+
4. Create an instance of the [Inspector API](./sdk-inspector.rst) by passing in the [ETDump](./sdk-etdump.md) you have sourced from the runtime along with the optionally generated [ETRecord](./sdk-etrecord.rst) from step 1 or execute the following command in your terminal to display the profiling data table.
112+
```bash
113+
python examples/apple/coreml/scripts/inspector_cli.py --etdump_path etdump.etdp --etrecord_path mv3_coreml.bin
114+
```
115+
116+
86117
## Deploying and running on a device
87118

88119
**Running the Core ML delegated Program in the Demo iOS App**:
@@ -92,37 +123,35 @@ cd executorch
92123

93124
3. Complete the [Final Steps](demo-apps-ios.md#final-steps) section of the tutorial to build and run the demo app.
94125

95-
<br>**Running the Core ML delegated Program in your own App**
96-
1. Build **Core ML** delegate. The following will create a `executorch.xcframework` in the `cmake-out` directory.
126+
<br>**Running the Core ML delegated Program in your App**
127+
1. Build frameworks, running the following will create a `executorch.xcframework` and `coreml_backend.xcframework` in the `cmake-out` directory.
97128
```bash
98129
cd executorch
99130
./build/build_apple_frameworks.sh --Release --coreml
100131
```
101132
2. Create a new [Xcode project](https://developer.apple.com/documentation/xcode/creating-an-xcode-project-for-an-app#) or open an existing project.
102133

103-
3. Drag the `executorch.xcframework` generated from Step 2 to Frameworks.
134+
3. Drag the `executorch.xcframework` and `coreml_backend.xcframework` generated from Step 2 to Frameworks.
104135

105136
4. Go to the project's [Build Phases](https://developer.apple.com/documentation/xcode/customizing-the-build-phases-of-a-target) - Link Binaries With Libraries, click the + sign, and add the following frameworks:
106137
```
107-
- executorch.xcframework
108-
- coreml_backend.xcframework
109-
- Accelerate.framework
110-
- CoreML.framework
111-
- libsqlite3.tbd
138+
executorch.xcframework
139+
coreml_backend.xcframework
140+
Accelerate.framework
141+
CoreML.framework
142+
libsqlite3.tbd
112143
```
113144
5. Add the exported program to the [Copy Bundle Phase](https://developer.apple.com/documentation/xcode/customizing-the-build-phases-of-a-target#Copy-files-to-the-finished-product) of your Xcode target.
114145

115-
6. Please follow the [running a model](running-a-model-cpp-tutorial.md) tutorial to integrate the code for loading a ExecuTorch program.
146+
6. Please follow the [running a model](./running-a-model-cpp-tutorial.md) tutorial to integrate the code for loading an ExecuTorch program.
116147

117148
7. Update the code to load the program from the Application's bundle.
118149
``` objective-c
119150
using namespace torch::executor;
120151

121152
NSURL *model_url = [NBundle.mainBundle URLForResource:@"mv3_coreml_all" extension:@"pte"];
122153

123-
Result<util::FileDataLoader> loader =
124-
util::FileDataLoader::from(model_url.path.UTF8String);
125-
154+
Result<util::FileDataLoader> loader = util::FileDataLoader::from(model_url.path.UTF8String);
126155
```
127156
128157
8. Use [Xcode](https://developer.apple.com/documentation/xcode/building-and-running-an-app#Build-run-and-debug-your-app) to deploy the application on the device.

‎examples/apple/coreml/README.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Examples
22

3-
This directory contains scripts and other helper utilities to illustrate an end-to-end workflow to run a **Core ML** delegated `torch.nn.module` with the **ExecuTorch** runtime.
3+
This directory contains scripts and other helper utilities to illustrate an end-to-end workflow to run a Core ML delegated `torch.nn.module` with the ExecuTorch runtime.
44

55

66
## Directory structure
@@ -13,7 +13,7 @@ coreml
1313

1414
## Using the examples
1515

16-
We will walk through an example model to generate a **Core ML** delegated binary file from a python `torch.nn.module` then we will use the `coreml/executor_runner` to run the exported binary file.
16+
We will walk through an example model to generate a Core ML delegated binary file from a python `torch.nn.module` then we will use the `coreml_executor_runner` to run the exported binary file.
1717

1818
1. Following the setup guide in [Setting Up ExecuTorch](https://pytorch.org/executorch/stable/getting-started-setup)
1919
you should be able to get the basic development environment for ExecuTorch working.
@@ -27,40 +27,50 @@ cd executorch
2727

2828
```
2929

30-
3. Run the export script to generate a **Core ML** delegated binary file.
30+
3. Run the export script to generate a Core ML delegated binary file.
3131

3232
```bash
3333
cd executorch
3434

3535
# To get a list of example models
3636
python3 -m examples.portable.scripts.export -h
3737

38-
# Generates ./add_coreml_all.pte file if successful.
38+
# Generates add_coreml_all.pte file if successful.
3939
python3 -m examples.apple.coreml.scripts.export --model_name add
4040
```
4141

42-
4. Once we have the **Core ML** delegated model binary (pte) file, then let's run it with the **ExecuTorch** runtime using the `coreml_executor_runner`.
42+
4. Run the binary file using the `coreml_executor_runner`.
4343

4444
```bash
4545
cd executorch
4646

4747
# Builds the Core ML executor runner. Generates ./coreml_executor_runner if successful.
4848
./examples/apple/coreml/scripts/build_executor_runner.sh
4949

50-
# Run the Core ML delegate model.
50+
# Run the delegated model.
5151
./coreml_executor_runner --model_path add_coreml_all.pte
5252
```
5353

5454
## Frequently encountered errors and resolution.
55-
- The `examples.apple.coreml.scripts.export` could fail if the model is not supported by the **Core ML** backend. The following models from the examples models list (` python3 -m examples.portable.scripts.export -h`)are currently supported by the **Core ML** backend.
55+
- The `examples.apple.coreml.scripts.export` could fail if the model is not supported by the Core ML backend. The following models from the examples models list (` python3 -m examples.portable.scripts.export -h`) are currently supported by the Core ML backend.
5656

57-
```
57+
```text
5858
add
5959
add_mul
60+
dl3
61+
edsr
62+
emformer_join
63+
emformer_predict
64+
emformer_transcribe
65+
ic3
6066
ic4
6167
linear
68+
llama2
69+
llava_encoder
70+
mobilebert
6271
mul
6372
mv2
73+
mv2_untrained
6474
mv3
6575
resnet18
6676
resnet50

0 commit comments

Comments
 (0)