Skip to content

Remove buck dependency from CMake build #8268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
GregoryComer opened this issue Feb 6, 2025 · 1 comment
Open

Remove buck dependency from CMake build #8268

GregoryComer opened this issue Feb 6, 2025 · 1 comment
Assignees
Labels
module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch module: user experience Issues related to reducing friction for users triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@GregoryComer
Copy link
Member

GregoryComer commented Feb 6, 2025

🚀 The feature, motivation and pitch

CMake builds currently rely on buck under the hood to generate source lists for each target. Buck is not available on all platforms and is the source of friction when building ExecuTorch. We've discussed and are largely aligned on the goal to remove the buck dependency from CMake, either by splitting source lists into a format that can be shared between buck and CMake, or by globbing.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @larryliu0820 @mergennachin @byjlw

@GregoryComer GregoryComer added module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch module: user experience Issues related to reducing friction for users labels Feb 6, 2025
@github-project-automation github-project-automation bot moved this to To triage in ExecuTorch DevX Feb 6, 2025
@GregoryComer GregoryComer moved this from To triage to Ready in ExecuTorch DevX Feb 6, 2025
@digantdesai digantdesai added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 6, 2025
@mergennachin mergennachin moved this from Ready to Backlog in ExecuTorch DevX Feb 6, 2025
@swolchok swolchok self-assigned this Feb 8, 2025
@swolchok
Copy link
Contributor

swolchok commented Feb 8, 2025

Here's my plan to execute on this. PR counts are estimates just to give an idea of what to expect.

  1. Create a PyTorch-like build_variables.bzl that is isomorphic to the current contents of executorch_srcs.cmake. Use (i.e., copy/paste/modify into ExecuTorch repo; it's short enough) PyTorch's append_filelist CMake function to ingest it. At CMake time, enforce that the filelists in build_variables.bzl are identical to the ones in the Buck-generated executorch_srcs.cmake. (1-2 PRs)
  2. Make build_variables.bzl the canonical way that the Buck-driven build finds all files it mentions. This will entail refactoring/splitting the filelists it contains to reflect Buck's finer-grained view of the world and then including (very simple) Starlark code in build_variables.bzl that reassembles the CMake-isomorphic set of filelists. (probably 1 PR per affected Buck target, many in total)
  3. Switch over from executorch_srcs.cmake to the filelists from build_variables.bzl and stop running Buck during builds. (1 PR)
  4. Clean up OSS-only Buck artifacts (CI scripts and workflows, shim files (keeping these as we want to run buck2 in CI to ease internal import process), installation, etc.). (small number of PRs)
  5. Clean up executorch_srcs_from_build_variables_bzl.cmake; directly produce CMake variables using append_filelist rather than writing out a .cmake file and reading it back in. (0-1 PR)
  6. (out of scope) Make the CMake target graph finer-grained and closer to the internal Buck graph.

If we wanted to be extremely aggressive, we could move (2) after (4), but I don't see any particular reason to rush to remove Buck from the process before the replacement setup is properly "livable".

swolchok added a commit that referenced this issue Feb 8, 2025
First step of #8268.

ghstack-source-id: b29aa49
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 9, 2025
First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

ghstack-source-id: 07b627b
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 10, 2025
First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

ghstack-source-id: 91476c9
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 10, 2025
First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

ghstack-source-id: b2ffb3b
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 10, 2025
First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

ghstack-source-id: dc50258
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 12, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 1608805
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 12, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 2c820ae
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 12, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 8d1e2c0
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 19, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 29773a8
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 22, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 89544c6
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 24, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: ca0a725
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 24, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 2152444
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 25, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: d575962
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
swolchok added a commit that referenced this issue Feb 25, 2025
First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

ghstack-source-id: c4195ce
ghstack-comment-id: 2644414497
Pull Request resolved: #8326
swolchok added a commit that referenced this issue Feb 25, 2025
Making the structure in this file mirror buck's worldview more closely when it makes sense. #8268

ghstack-source-id: 22a10ed
ghstack-comment-id: 2652360999
Pull Request resolved: #8398
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch module: user experience Issues related to reducing friction for users triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
Status: Backlog
Development

No branches or pull requests

3 participants