Remove usage of external flatc in builds and scripts #9306

jathu · 2025-03-15T20:51:42Z

Summary

Remove the dependence of EXECUTORCH_BUILD_HOST_TARGETS since we always build flatc for the host now
Turn on EXECUTORCH_BUILD_FLATC if FLATC_EXECUTABLE is not set
Build flatc from source for Apple builds

Test Plan

Now macOS/Linux targeting macOS/Linux/iOS/Android should all build flatc from source:

$ rm -rf pip-out && ./install_executorch.sh
$ rm -rf cmake-out* && ./build/build_apple_frameworks.sh
$ rm -rf cmake-out* && ./build/build_android_library.sh

cc @larryliu0820 @lucylq

pytorch-bot · 2025-03-15T20:51:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9306

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 3ec8bd8 with merge base 4c54bab ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

trunk / test-arm-backend-delegation / linux-job (gh) (#9399)
RuntimeError: Command docker exec -t c558af1ed82f70b61c8fcb178370b56909bb3be33979e9a139510df901136592 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zingo

Cheked/reviewed the Arm files, thanks for fixing this. I hope we can use the same solution for flatcc down the line if this work out good. Great work getting this annoyance fixed.

mergennachin · 2025-03-17T19:35:07Z

@jathu are the CI failures related?

jathu · 2025-03-19T17:07:08Z

@jathu are the CI failures related?

@mergennachin should be good now. The two failing arm tests are failing on main

zingo · 2025-03-19T17:57:27Z

https://github.com/pytorch/executorch/actions/runs/13950476776/job/39048452046?pr=9306

Has not even run at all (or logs are missing?) there seem to be some major flakeness just running the arm test e.g. the CI seem to not start them as expected :(

The other job run but had a strange error, I retriggered both

mergennachin

Shall we delete EXECUTORCH_BUILD_HOST_TARGETS option altogether from the repo?

There are still some remaining references un CmakeLists.txt and Utils.cmake

jathu · 2025-03-19T18:05:17Z

@mergennachin yes, #9404

@larryliu0820

### Summary * Remove the dependence of `EXECUTORCH_BUILD_HOST_TARGETS` since we always build flatc for the host now * Turn on `EXECUTORCH_BUILD_FLATC` if `FLATC_EXECUTABLE` is not set * Build flatc from source for Apple builds ### Test Plan Now macOS/Linux targeting macOS/Linux/iOS/Android should all build flatc from source: ``` $ rm -rf pip-out && ./install_executorch.sh $ rm -rf cmake-out* && ./build/build_apple_frameworks.sh $ rm -rf cmake-out* && ./build/build_android_library.sh ``` cc @larryliu0820 @lucylq

@larryliu0820

### Summary * Remove the dependence of `EXECUTORCH_BUILD_HOST_TARGETS` since we always build flatc for the host now * Turn on `EXECUTORCH_BUILD_FLATC` if `FLATC_EXECUTABLE` is not set * Build flatc from source for Apple builds ### Test Plan Now macOS/Linux targeting macOS/Linux/iOS/Android should all build flatc from source: ``` $ rm -rf pip-out && ./install_executorch.sh $ rm -rf cmake-out* && ./build/build_apple_frameworks.sh $ rm -rf cmake-out* && ./build/build_android_library.sh ``` cc @larryliu0820 @lucylq

@mergennachin

### Summary Removes EXECUTORCH_BUILD_HOST_TARGETS per @mergennachin [comment](#9306 (review)). Fixes #9404 ### Test plan Ensure executorch and cadence builds successfully: ``` ./install_executorch.sh mkdir cmake-out cmake -DCMAKE_TOOLCHAIN_FILE=backends/cadence/cadence.cmake \ -DCMAKE_INSTALL_PREFIX=cmake-out \ -DCMAKE_BUILD_TYPE=Debug \ -DPYTHON_EXECUTABLE=python3 \ -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \ -DEXECUTORCH_BUILD_HOST_TARGETS=ON \ -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=OFF \ -DEXECUTORCH_BUILD_PTHREADPOOL=OFF \ -DEXECUTORCH_BUILD_CPUINFO=OFF \ -Bcmake-out . cmake --build cmake-out -j8 --target install --config Debug cmake -DCMAKE_BUILD_TYPE=Debug \ -DCMAKE_TOOLCHAIN_FILE=examples/backends/cadence.cmake \ -DCMAKE_PREFIX_PATH=cmake-out \ -DMODEL_PATH=add.pte \ -DNXP_SDK_ROOT_DIR=/home/user_name/nxp-sdk \ -DNN_LIB_BASE_DIR=/home/user_name/nnlib-hifi4 \ -Bcmake-out/examples/cadence \ examples/cadence cmake --build cmake-out/examples/cadence -j8 -t cadence_executorch_example ls cmake-xt/*.bin ``` * Test plan assumes nxp-sdk and nnlib-hifir are in user's home directory and Xtensa compiler is available (env vars set). See [cadence docs](https://github.com/pytorch/executorch/blob/main/docs/source/backends-cadence.md) * Since these are modifications to the build system, I leave it to discretion if CI builds provide enough risk protection to merge. * Happy to write tests if needed - I'll need a little direction here if needed. #9306 ran `./install_executorch.sh`, `build_apple_frameworks.sh` and `build_android_library.sh`. --------- Co-authored-by: Mergen Nachin <[email protected]>

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 15, 2025

jathu added module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch ciflow/trunk labels Mar 15, 2025

jathu added this to ExecuTorch DevX Mar 15, 2025

github-project-automation bot moved this to To triage in ExecuTorch DevX Mar 15, 2025

jathu force-pushed the jathu/remove-flatc--build-apple branch 3 times, most recently from 8c9b1c8 to 609c745 Compare March 15, 2025 21:19

jathu changed the title ~~Build flatc for Apple builds from source~~ Remove usage of external flatc in builds and scripts Mar 15, 2025

jathu force-pushed the jathu/remove-flatc--build-apple branch 2 times, most recently from 31830f5 to 2a5d7d7 Compare March 16, 2025 18:12

jathu moved this from To triage to In progress in ExecuTorch DevX Mar 16, 2025

zingo approved these changes Mar 17, 2025

View reviewed changes

jathu marked this pull request as ready for review March 17, 2025 14:44

jathu requested review from cccclai, metascroy, cymbalrush, mergennachin, GregoryComer, kirklandsign, tarun292, digantdesai and shoumikhin as code owners March 17, 2025 14:44

jathu force-pushed the jathu/remove-flatc--build-apple branch from 2a5d7d7 to ac6b9fd Compare March 17, 2025 14:45

jathu requested review from larryliu0820 and swolchok March 17, 2025 14:51

jathu force-pushed the jathu/remove-flatc--build-apple branch 2 times, most recently from 8acf205 to aa73a64 Compare March 17, 2025 22:20

jathu added the release notes: build Changes related to build, including dependency upgrades, build flags, optimizations, etc. label Mar 17, 2025

jathu force-pushed the jathu/remove-flatc--build-apple branch 2 times, most recently from 4943be0 to 4e3f774 Compare March 18, 2025 19:04

jathu marked this pull request as draft March 18, 2025 19:58

jathu force-pushed the jathu/remove-flatc--build-apple branch 2 times, most recently from bfb21f9 to e2c853d Compare March 18, 2025 21:44

jathu marked this pull request as ready for review March 18, 2025 23:04

jathu force-pushed the jathu/remove-flatc--build-apple branch from e2c853d to a40e1e4 Compare March 18, 2025 23:47

remove the ability to bring flatc

3ec8bd8

jathu force-pushed the jathu/remove-flatc--build-apple branch from a40e1e4 to 3ec8bd8 Compare March 19, 2025 15:31

mergennachin approved these changes Mar 19, 2025

View reviewed changes

jathu mentioned this pull request Mar 19, 2025

Remove EXECUTORCH_BUILD_HOST_TARGETS #9404

Closed

jathu merged commit bcea345 into main Mar 19, 2025
257 of 259 checks passed

jathu deleted the jathu/remove-flatc--build-apple branch March 19, 2025 18:26

github-project-automation bot moved this from In progress to Done in ExecuTorch DevX Mar 19, 2025

jathu mentioned this pull request Mar 19, 2025

Remove the ability to bring your own flatc #9145

Closed

bvanderhaar mentioned this pull request Apr 21, 2025

refactor no longer needed EXECUTORCH_BUILD_HOST_TARGETS #10320

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove usage of external flatc in builds and scripts #9306

Remove usage of external flatc in builds and scripts #9306

jathu commented Mar 15, 2025 •

edited

Loading

pytorch-bot bot commented Mar 15, 2025 •

edited

Loading

zingo left a comment

mergennachin commented Mar 17, 2025

jathu commented Mar 19, 2025 •

edited

Loading

zingo commented Mar 19, 2025 •

edited

Loading

mergennachin left a comment

jathu commented Mar 19, 2025

Remove usage of external flatc in builds and scripts #9306

Remove usage of external flatc in builds and scripts #9306

Conversation

jathu commented Mar 15, 2025 • edited Loading

Summary

Test Plan

pytorch-bot bot commented Mar 15, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9306

✅ You can merge normally! (1 Unrelated Failure)

zingo left a comment

Choose a reason for hiding this comment

mergennachin commented Mar 17, 2025

jathu commented Mar 19, 2025 • edited Loading

zingo commented Mar 19, 2025 • edited Loading

mergennachin left a comment

Choose a reason for hiding this comment

jathu commented Mar 19, 2025

jathu commented Mar 15, 2025 •

edited

Loading

pytorch-bot bot commented Mar 15, 2025 •

edited

Loading

jathu commented Mar 19, 2025 •

edited

Loading

zingo commented Mar 19, 2025 •

edited

Loading