Skip to content

A warning/note message for reminding people to clone submodule after the first clone. #7243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mergennachin opened this issue Dec 9, 2024 · 6 comments
Closed

A warning/note message for reminding people to clone submodule after the first clone. #7243

mergennachin opened this issue Dec 9, 2024 · 6 comments
Assignees
@larryliu0820
Labels
module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch rfc Request for comment and feedback on a post, proposal, etc. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone
0.6.0

Comments

@mergennachin
Copy link
Contributor

mergennachin commented Dec 9, 2024 •
edited by pytorch-bot bot
Loading

🚀 The feature, motivation and pitch

At first trial, they forgot to clone with submodule → got error cannot find “gflags“ → they need to remember to clone submodule.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @larryliu0820
@mergennachin mergennachin moved this to To triage in ExecuTorch DevX Dec 9, 2024
@mcr229 mcr229 added feature rfc Request for comment and feedback on a post, proposal, etc. labels Jan 14, 2025
@GregoryComer
Copy link
Member

GregoryComer commented Jan 17, 2025 •
edited
Loading

This may be more work than we want to do, but I've seen some projects lazy-clone submodules at build time. That also carries the advantage of only pulling submodules that we actually need. If you aren't building all of the various NPU backends, for example, you don't have to pull them down. I believe CMake also has some level of git integration to help with this.

Another closely related problem I've seen is forgetting to update submodules after pulling the latest ET. This can lead to weird build errors and sometimes takes me a few minutes to realize what's wrong. I'm not sure if there's a good way to detect this, though.

@larryliu0820 larryliu0820 mentioned this issue Feb 5, 2025
Merged
@larryliu0820 larryliu0820 self-assigned this Feb 5, 2025
@larryliu0820 larryliu0820 moved this from To triage to In progress in ExecuTorch DevX Feb 5, 2025
larryliu0820 added a commit that referenced this issue Feb 5, 2025
@larryliu0820
Update on "[BE] Remind users to update submodule"
fa242fb 
Summary: 

Fixing #7243

As titled. This adds messages when we suspect the user forgot
to update git submodule or cleanup the CMake cache.

Test Plan: 

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following error messages

```
  Command failed:
  From load at implicit location

  Caused by:
      File not found: `prelude//prelude.bzl`


  CMake Error at build/Utils.cmake:230 (message):
    executorch: source list generation failed
  Call Stack (most recent call first):
    CMakeLists.txt:386 (extract_sources)


  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "<string>", line 643, in run
    File "/home/larryliu/miniconda3/envs/executorch/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 388, in spawn
      spawn(cmd, search_path, dry_run=self.dry_run)
    File "/home/larryliu/miniconda3/envs/executorch/lib/python3.11/site-packages/setuptools/_distutils/spawn.py", line 68, in spawn
      raise DistutilsExecError(f"command {cmd!r} failed with exit code {exitcode}")
  distutils.errors.DistutilsExecError: command '/home/larryliu/miniconda3/envs/executorch/bin/cmake' failed with exit code 1
...
  Exception: command '/home/larryliu/miniconda3/envs/executorch/bin/cmake' failed with exit code 1
  ExecuTorch: Either CMake cache is outdated or git submodules are not synced.
  ExecuTorch: Please run the following before retry:
  ExecuTorch:    ./install_executorch.sh --clean
  ExecuTorch:    git submodule update --init --recursive

  error: subprocess-exited-with-error
```

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this issue Feb 5, 2025
@larryliu0820
Update on "[BE] Remind users to update submodule"
117ae6e 
Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh
```

See the following error:


Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this issue Feb 5, 2025
@larryliu0820
Update on "[BE] Remind users to update submodule"
e642591 
Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh --clean
./install_executorch.sh
```

See the following error:

```
  2025-02-05 12:08:56,889 [ExecuTorch] ERROR: Failed to query buck for sources. Failed command:

     buck2 cquery inputs(deps('//runtime/executor:program'))

  This is likely due to missing git submodules or outdated CMake cache. Please run the following before retry:

      ./install_executorch.sh --clean
      git submodule update --init --recursive

```
Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this issue Feb 5, 2025
@larryliu0820
Update on "[BE] Remind users to update submodule"
c8583b4 
Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh --clean
./install_executorch.sh
```

See the following error:

```
  2025-02-05 12:08:56,889 [ExecuTorch] ERROR: Failed to query buck for sources. Failed command:

     buck2 cquery inputs(deps('//runtime/executor:program'))

  This is likely due to missing git submodules or outdated CMake cache. Please run the following before retry:

      ./install_executorch.sh --clean
      git submodule update --init --recursive

```
Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]
larryliu0820 added a commit that referenced this issue Feb 5, 2025
@larryliu0820
[BE] Remind users to update submodule (#8203)
35369e0 
* [BE] Remind users to update submodule

Summary: As titled. This adds messages when we suspect the user forgot
to update git submodule or cleanup the CMake cache.

Test Plan: See the following error messages

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]

* Update on "[BE] Remind users to update submodule"


Summary: 

Fixing #7243

As titled. This adds messages when we suspect the user forgot
to update git submodule or cleanup the CMake cache.

Test Plan: 

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following error messages

```
  Command failed:
  From load at implicit location

  Caused by:
      File not found: `prelude//prelude.bzl`


  CMake Error at build/Utils.cmake:230 (message):
    executorch: source list generation failed
  Call Stack (most recent call first):
    CMakeLists.txt:386 (extract_sources)


  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "<string>", line 643, in run
    File "/home/larryliu/miniconda3/envs/executorch/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 388, in spawn
      spawn(cmd, search_path, dry_run=self.dry_run)
    File "/home/larryliu/miniconda3/envs/executorch/lib/python3.11/site-packages/setuptools/_distutils/spawn.py", line 68, in spawn
      raise DistutilsExecError(f"command {cmd!r} failed with exit code {exitcode}")
  distutils.errors.DistutilsExecError: command '/home/larryliu/miniconda3/envs/executorch/bin/cmake' failed with exit code 1
...
  Exception: command '/home/larryliu/miniconda3/envs/executorch/bin/cmake' failed with exit code 1
  ExecuTorch: Either CMake cache is outdated or git submodules are not synced.
  ExecuTorch: Please run the following before retry:
  ExecuTorch:    ./install_executorch.sh --clean
  ExecuTorch:    git submodule update --init --recursive

  error: subprocess-exited-with-error
```

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]

* Update on "[BE] Remind users to update submodule"


Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh
```

See the following error:


Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]

* Update on "[BE] Remind users to update submodule"


Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh --clean
./install_executorch.sh
```

See the following error:

```
  2025-02-05 12:08:56,889 [ExecuTorch] ERROR: Failed to query buck for sources. Failed command:

     buck2 cquery inputs(deps('//runtime/executor:program'))

  This is likely due to missing git submodules or outdated CMake cache. Please run the following before retry:

      ./install_executorch.sh --clean
      git submodule update --init --recursive

```
Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]

* Update on "[BE] Remind users to update submodule"


Summary: 

Fixing #7243

As titled. This PR adds 2 things:
1. `check_and_update_submodule()` check if required submodule folders exist and at least contains a CMakeLists.txt file.
2. Give useful error when the submodule is corrupted (most likely missing a file, caused by submodule out of sync).

Test Plan: 

1. Test if we can still install, if submodule is not updated

```
rm -rf third-party/prelude
./install_executorch.sh
```
See the following log

```
...
Processing /data/users/larryliu/executorch
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  2025-02-05 11:28:46,430 [ExecuTorch] WARNING: Some required submodules are missing. Updating submodules...
  Submodule path 'third-party/prelude': checked out '851d3f09c452937fc5adef27e2c50f7f304f1646'
  2025-02-05 11:28:46,748 [ExecuTorch] INFO: All required submodules are present.
  2025-02-05 11:28:47,018 [ExecuTorch] INFO: running dist_info
...
```
This proves that we can update submodule for the user.

2. Test if we can give useful error message, if we are missing a file.

```
rm third-party/gflags/src/gflags.cc
./install_executorch.sh --clean
./install_executorch.sh
```

See the following error:

```
  2025-02-05 12:08:56,889 [ExecuTorch] ERROR: Failed to query buck for sources. Failed command:

     buck2 cquery inputs(deps('//runtime/executor:program'))

  This is likely due to missing git submodules or outdated CMake cache. Please run the following before retry:

      ./install_executorch.sh --clean
      git submodule update --init --recursive

```
Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D69156975](https://our.internmc.facebook.com/intern/diff/D69156975)

[ghstack-poisoned]
@mergennachin
Copy link
Contributor Author

mergennachin commented Feb 5, 2025 •
edited
Loading

I just did a fresh clone, pulled in your PR (#8229) and tried install_executorch.sh, before the git submodule init/update steps.

Here's my error: https://gist.github.com/mergennachin/e218cfe00b4809a14d2a2ffe4abd61f4

In your Test Plan, you didn't actually test a fresh clone scenario.

@mergennachin mergennachin added this to the 0.6.0 milestone Feb 6, 2025
@digantdesai digantdesai added triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch labels Feb 7, 2025
@larryliu0820
Copy link
Contributor

In your Test Plan, you didn't actually test a fresh clone scenario.

@mergennachin I believe this is fixed. Can you retry?

@larryliu0820
Copy link
Contributor

Closing for now feel free to reopen

@github-project-automation github-project-automation bot moved this from In progress to Done in ExecuTorch DevX Feb 10, 2025
@mergennachin mergennachin reopened this Feb 11, 2025
@github-project-automation github-project-automation bot moved this from Done to Backlog in ExecuTorch DevX Feb 11, 2025
@mergennachin
Copy link
Contributor Author

mergennachin commented Feb 11, 2025 •
edited
Loading

@larryliu0820

Here's my repro:

  • I cloned executorch repo
  • Created a new conda environment and activate
  • DO NOT run git submodule commands.
  • Run ./install_requirements.sh but it is failing with the same error.

@mergennachin
Copy link
Contributor Author

Ah sorry, should've run ./install_executorch.sh

@github-project-automation github-project-automation bot moved this from In progress to Done in ExecuTorch DevX Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@larryliu0820 larryliu0820

Labels
module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch rfc Request for comment and feedback on a post, proposal, etc. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
ExecuTorch DevX
Status: Done
Milestone
0.6.0
Development

No branches or pull requests
5 participants
@digantdesai @mergennachin @larryliu0820 @GregoryComer @mcr229