Qualcomm AI Engine Direct - Enable zero copy feature

[shewu-quic]

Summary:

• Add argument "shared_buffer" into compiler_spec, qnn_executor_runner and test scripts
  • Actually, shared_buffer should be a runtime option since user are responsible to allocate memory for tensors on device. But it seems to have no way to set the runtime option to QnnBackend. Therefore, we put it to compile_spec for now.
• Implement SharedBuffer to allocate and free RPC memory
• Add QnnMemManger to register shared buffer for tensor
  • During exection time, we will register memory of tensor data for QNN. And we will deregister them during destruction time of QnnBackend
• Add two API void* QnnExecuTorchAllocCustomMem(size_t bytes, size_t alignment) and void QnnExecuTorchFreeCustomMem(void* buffer_ptr) to allocate RPC memory with SharedBuffer
  • Users are responsible to allocate "enough" tensor bytes, and set alignment as MemoryAllocator::kDefaultAlignment. See runtime/core/memory_allocator.h.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/2531

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Unrelated Failure

As of commit e541fb4 with merge base 7f96f5a ():

NEW FAILURES - The following jobs have failed:

• pull / test-llama-runner-linux (fp16, buck2) / linux-job (gh)
  RuntimeError: Command docker exec -t db46c3b5bef390f08c7ff687f399c4894c50543c6506260b8621a219b4355d84 /exec failed with exit code 2
• pull / test-llama-runner-linux (fp16, cmake) / linux-job (gh)
  RuntimeError: Command docker exec -t 0f89a2581ac9c8af2d4b9b16dc2cb8f8ffbd6d0ef6b9639d33772ea0ed13ffb5 /exec failed with exit code 2
• pull / test-llama-runner-linux (fp32, buck2) / linux-job (gh)
  RuntimeError: Failed to compile /tmp/tmpah9x4wjy/data.json to /tmp/tmpah9x4wjy/data.pte. Set ET_EXIR_SAVE_FLATC_INPUTS_ON_FAILURE=1 to save input files on failure.
• pull / test-llama-runner-linux (fp32, cmake) / linux-job (gh)
  RuntimeError: Failed to compile /tmp/tmpirb3mwpr/data.json to /tmp/tmpirb3mwpr/data.pte. Set ET_EXIR_SAVE_FLATC_INPUTS_ON_FAILURE=1 to save input files on failure.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos (buck2) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 5

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey do you mind rebasing again? As #2506 was landed and there is a merge conflict

[shewu-quic]

Thanks for your reminder.
I have rebased my PR.
But I find some errors for quantized test, it seems fail to get_fake_program in backend_api.py.
Do you have any ideas about it or I miss something?

Test in pytorch/main branch.

python3 backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a4w_conv2d -b build_android -s $DEVICE -H $HOST -m $MODEL -r executorch -a unit_test 

[cccclai]

Oh that might be related to a recent change in #2502. cc: @lucylq

[lucylq]

Oh that might be related to a recent change in #2502. cc: @lucylq

Thanks - taking a look!

[lucylq]

Hey @shewu-quic, do you mind rebasing and trying again? Added a fallback here: #2564

[shewu-quic]

Done. Thanks for your help. :)

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai merged this pull request in a531ca5.