Add dynamic shape support to CoreML #9094
Conversation
Summary: Dynamic shape currently lead to runtime errors in CoreML. The AOT flow prepares the CoreML pte to work with dynamic shapes. These inputs are correctly evaluated by the CoreML model, but a runtime error occurs when copying the model outputs back to ET. This is because the model outputs have shape that is smaller than the outputArg MultiArray and the copy fails. This PR resizes the output args to account for dynamic shapes in the ET CoreML delegate. Note that outputBackings appear to fail in the dynamic case, likely because the outputBacking tensor is bigger than it should be. Ideally CoreML would not fail as long as the outputBacking had enough storage to hold the output since we don't know the expected size until after the model runs. The output is resized after the model is run anyway. In any case, dynamic shapes take the "ignoreOutputBackings" route.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9094
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 2da75d5 with merge base db4a500 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
@metascroy has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
Thanks - lint error needs to be fixed and it will be good to have a test coverage. |
| @@ -903,6 +903,10 @@ if(EXECUTORCH_BUILD_EXECUTOR_RUNNER) | |||
| endif() | |||
| endif() | |||
|
|
|||
| if(EXECUTORCH_BUILD_COREML) | |||
| list(APPEND _executor_runner_libs coremldelegate) | |||
There was a problem hiding this comment.
Is it for the portable executor runner? Is it needed?
There was a problem hiding this comment.
I can remove, it was just very convenient to have a C++ runner that works with CoreML pte files for debugging the existing crash.
Will add |
Added new tests for resize. In doing so, I noticed one of the existing tests is broken on main (diabled it in this PR). We don't run these tests in CI, but I added an issue to enable them (#9115). |
|
@metascroy has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@shoumikhin can you take a look at the error in building the benchmark app in the CI here. It's not clear to me what the issue is. Note that the frameworks build correctly. |
metascroy
added
the
module: coreml
|
@pytorchbot label "topic: not user facing" |
Differential Revision: D70904915 Pull Request resolved: pytorch#9094
Summary:
Dynamic shape currently lead to runtime errors in CoreML.
The AOT flow prepares the CoreML pte to work with dynamic shapes. These inputs are correctly evaluated by the CoreML model, but a runtime error occurs when copying the model outputs back to ET.
This is because the model outputs have shape that is smaller than the outputArg MultiArray and the copy fails.
This PR resizes the output args to account for dynamic shapes in the ET CoreML delegate.
Note that outputBackings appear to fail in the dynamic case, likely because the outputBacking tensor is bigger than it should be. Ideally CoreML would not fail as long as the outputBacking had enough storage to hold the output since we don't know the expected size until after the model runs. The output is resized after the model is run anyway.
In any case, dynamic shapes take the "ignoreOutputBackings" route.
cc @kimishpatel @YifanShenSZ @cymbalrush