[Build] fix build issues for native Windows. #4681

python3kgae · 2024-08-13T03:18:08Z

Add install_requirements.ps1 for build on native Windows.

Also fix issues block build on native Windows.

Get the Windows version of buck2 for Windows.
Check the path based on the build configuration for Windows in setup.py.
Avoid generating 'def flat.exe()' which is illegal python.
Add the Windows version of kernel_link_options.
Add links of pthreadpool and cpuinfo for custom_ops_aot_lib.
Define ssize_t for Windows.

Still have one link issue tracked with #4659.

For #4661

pytorch-bot · 2024-08-13T03:18:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4681

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures

As of commit 1b8899c with merge base f887d72 ():

NEW FAILURES - The following jobs have failed:

pull / test-llama-runner-linux (fp32, buck2, portable) / linux-job (gh)
RuntimeError: Command docker exec -t 007eed95632dd1c485687bc868cd04fc48325219d6188733b822da5063d2e75a /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, buck2, xnnpack+custom) / linux-job (gh)
RuntimeError: Command docker exec -t 4d9cd537001a4281f6ced99e9e4d1b82e2f15243a8722a7d147ae8ea654dc6be /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, buck2, xnnpack+custom+qe) / linux-job (gh)
RuntimeError: Command docker exec -t e87aeea7e2272c772de0142480e548a1e08ad44fd9ec623e09327a94bed7c214 /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, cmake, portable) / linux-job (gh)
RuntimeError: Command docker exec -t 986b1ffb3f6cce96b12fc13e166efed015bb9412ee8e3d1d3b3011e92ac0f20e /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, cmake, xnnpack+custom) / linux-job (gh)
RuntimeError: Command docker exec -t b337d3e4bc07c394f7ddf0b038da21e5ad4cbb2789994da3406a0d5d044bae04 /exec failed with exit code 1
pull / test-llama-runner-linux (fp32, cmake, xnnpack+custom+qe) / linux-job (gh)
RuntimeError: Command docker exec -t a05270c98de59ed963537b8cb166a46757ab345399db5dc74162bb04355bccb1 /exec failed with exit code 1
pull / test-llama-runner-qnn-linux (fp32, cmake, qnn) / linux-job (gh)
RuntimeError: Command docker exec -t f03d94b2132bfd7bb7a31b4b2e4c15c213d154fdad789e9ccb4726366a4152db /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

build/resolve_buck.py

kirklandsign

Thank you! Is this PR tested and ready for review? Also for the install_requirements.ps1, it's for windows only?

extension/data_loader/file_data_loader.cpp

python3kgae · 2024-08-13T21:31:28Z

Thank you! Is this PR tested and ready for review? Also for the install_requirements.ps1, it's for windows only?

I didn't really do any real tests except make sure the getting-started-setup works under native Windows and WSL.
Will the CI pipeline test cover all tests or there're some tests must be running locally?

And yes, install_requirements.ps1 is for windows only.

dbort

Thank you for taking the time to track down and fix all of these issues! I'm generally happy with the approach, but I'd really like to find a way to avoid duplicating so much data and logic from install_requirements.sh.

Will the CI pipeline test cover all tests or there're some tests must be running locally?

Unfortunately the CI pipeline doesn't cover install_requirements.sh right now. If you can, please try running it on a linux and/or mac with your changes. If you don't have access to either, let us know, and we can help testing.

dbort · 2024-08-14T19:23:23Z

runtime/core/exec_aten/util/tensor_util.h

@@ -15,6 +15,11 @@
 #include <cstddef> // size_t
 #include <limits>

+#ifdef _WIN32


This shouldn't be necessary since platform/compiler.h has the same logic.

If this needed to happen because one of the headers between here and line 29 refer to ssize_t, then the header that needs it should also include compiler.h.

But in general, compiler.h is the only place in the ET code that is allowed to have system-specific checks like #ifdef _WIN32

runtime/core/portable_type/tensor_impl.h

setup.py

dbort · 2024-08-14T19:26:53Z

setup.py

@@ -508,6 +508,9 @@ def run(self):
                item for item in os.environ["CMAKE_BUILD_ARGS"].split(" ") if item
            ]

+        if os.name == "nt":


Please add a comment explaining what this does, and why it's only necessary on windows. If it would be safe to use this flag on all systems, I'd rather remove the nt check.

setup.py

dbort · 2024-08-14T19:46:59Z

setup.py

+    debug = os.environ.get("DEBUG", 0)
+    cfg = "Debug" if debug else "Release"


Rather than duplicating this logic, please hoist it (and the other example near line 435) into a common global function like get_build_type() -> str

Added get_build_type.

dbort · 2024-08-14T19:50:14Z

setup.py

-        )
+        if os.name == "nt":
+            ext_modules.append(
+                BuiltFile(


I'd rather extend BuiltFile to handle this situation in a common way, instead of manually repeating the pattern for every entry.

E.g., it could take an extra is_executable: bool = False param that could be set to true here, letting it know to add the .exe suffix on windows.

And we could embed a CMAKE_BUILD_TYPE (debug vs release) placeholder in the paths that is ignored on non-windows systems. How about

third-party/flatbuffers/%BUILD_TYPE%/flatc

Then, on windows systems we can replace %BUILD_TYPE% with get_build_type(), and on non-windows systems we can delete the string %BUILD_TYPE%/ from the path if present.

So this this entry could become

ext_modules.append( BuiltFile( "third-party/flatbuffers/%BUILD_TYPE%/flatc", "executorch/data/bin/", is_executable=True, ) )

And entries below can use the %BUILD_TYPE% placeholder, and use .* to match .so/.dll/.dylib on different systems.

extension/data_loader/file_data_loader.cpp

dbort · 2024-08-14T19:59:04Z

extension/data_loader/file_data_loader.cpp

@@ -17,7 +17,11 @@
 #include <fcntl.h>
 #include <sys/stat.h>
 #include <sys/types.h>
+#ifndef _WIN32


I really want to avoid _WIN32 checks in any of the code, but this is reasonable for now. The best solution would be to avoid using POSIX concepts and to just use the standard C APIs.

Suggested change

#ifndef _WIN32

// TODO: Rewrite FileDataLoader to use standard C FILE* instead of POSIX file

// descriptors, so that we don't need this _WIN32 check.

#ifndef _WIN32

extension/llm/custom_ops/CMakeLists.txt

python3kgae · 2024-08-14T20:39:38Z

Thank you for taking the time to track down and fix all of these issues! I'm generally happy with the approach, but I'd really like to find a way to avoid duplicating so much data and logic from install_requirements.sh.

Will the CI pipeline test cover all tests or there're some tests must be running locally?

Unfortunately the CI pipeline doesn't cover install_requirements.sh right now. If you can, please try running it on a linux and/or mac with your changes. If you don't have access to either, let us know, and we can help testing.

Thanks for the review.
To avoid duplicating install_requirements.sh, I’m considering rewriting it in portable Python. I’ll give it a try to see if it works.

I don’t have access to Linux or macOS, so I’ll need help testing it on those platforms once I’ve got install_requirements.py working

Added install_requirements.ps1 for build on native Windows. Also fix issues block build on native Windows. 1. Get the Windows version of buck2 for Windows. 2. Check the path based on the build configuration for Windows in setup.py. 3. Avoid generating 'def flat.exe()' which is illegal python. 4. Add the Windows version of kernel_link_options. 5. Add links of pthreadpool and cpuinfo for custom_ops_aot_lib. 6. Define ssize_t for Windows. Still have one link issue tracked with pytorch#4659. For pytorch#4661

python3kgae · 2024-08-21T02:17:50Z

Updated based on the comments.
Tested install_requirements.sh on WSL and install_requirements.bat on Windows.
Need help to test Linux and Mac.

dbort

This is looking great!

Consider splitting this into some smaller PRs. E.g. we can probably merge the setup.py stuff pretty quickly, but the FileDataLoader might take longer. And, in general, we prefer to have smaller self-contained PRs.

build/resolve_buck.py

dbort · 2024-08-22T18:49:39Z

build/pip_data_bin_init.py.in

@@ -21,7 +21,9 @@ def _find_executable_files_under(dir):
    for filename in os.listdir(dir):
        filepath = os.path.join(dir, filename)
        if os.path.isfile(filepath) and os.access(filepath, os.X_OK):
-            bin_names.append(filename)
+            # avoid def flat.exe() on windows.


Comments should begin with a capital letter. And I suggest making this a little more general.

Suggested change

# avoid def flat.exe() on windows.

# Remove .exe suffix on windows.

dbort · 2024-08-22T18:58:19Z

extension/data_loader/file_data_loader.cpp

@@ -17,7 +17,11 @@
 #include <fcntl.h>
 #include <sys/stat.h>
 #include <sys/types.h>
+#ifndef _WIN32


Heads up that #4760 merged recently, which changes this class to use pread(), which I think will break windows. It might make sense to write a separate WindowsFileDataLoader.

There would be a lot of duplicated code, though. If we needed to split into a windows-specific data loader, it might make sense to have an AbstractFileDataLoader with one pure-virtual protected method to do the actual read.

dbort · 2024-08-22T19:00:27Z

install_requirements.py

@@ -83,6 +83,10 @@ def python_is_compatible():
        print(f"Error: Unknown option {arg}")
        sys.exit(1)

+# Use ClangCL on Windows.


This comment repeats what the code does, and doesn't provide extra value. Either remove it, or (ideally) add a comment explaining why this logic is necessary. It'd also be helpful to explain what ClangCL is, since non-windows devs will not be familiar with it.

dbort · 2024-08-22T19:03:27Z

runtime/platform/compiler.h

+#ifndef _WIN32
+#include <sys/types.h> // TODO(T126923429): Include size_t, ssize_t
+#else
+#include <stddef.h>
+using ssize_t = ptrdiff_t;
+#endif


Your PR actually fixes the TODO, so we can remove it.

Suggested change

#ifndef _WIN32

#include <sys/types.h> // TODO(T126923429): Include size_t, ssize_t

#else

#include <stddef.h>

using ssize_t = ptrdiff_t;

#endif

// Define size_t and ssize_t.

#ifndef _WIN32

#include <sys/types.h>

#else

#include <stddef.h>

using ssize_t = ptrdiff_t;

#endif

dbort · 2024-08-22T23:59:53Z

setup.py

@@ -508,6 +559,8 @@ def run(self):
                item for item in os.environ["CMAKE_BUILD_ARGS"].split(" ") if item
            ]

+        build_args += ["--config", cfg]


Please add a comment explaining why this is necessary.

dbort · 2024-08-23T00:01:22Z

setup.py

+                "third-party/flatbuffers/%BUILD_TYPE%/",
+                "flatc",
+                "executorch/data/bin/",


Now that there are more than two params, please use kwargs for all params. Same for other BuiltFile uses below.

Suggested change

"third-party/flatbuffers/%BUILD_TYPE%/",

"flatc",

"executorch/data/bin/",

src_path="third-party/flatbuffers/%BUILD_TYPE%/",

src_name="flatc",

dst="executorch/data/bin/",

dbort · 2024-08-23T00:02:14Z

setup.py

-    def __init__(self, src: str, dst: str):
+    def __init__(
+        self,
+        src_path: str,


Please call this src_dir since it's specifically a directory. src_path sounds like it could still refer to the path to the source file.

dbort · 2024-08-23T00:04:13Z

setup.py

    ext_modules = []
    if ShouldBuild.flatc():
        ext_modules.append(
-            BuiltFile("third-party/flatbuffers/flatc", "executorch/data/bin/")
+            BuiltFile(
+                "third-party/flatbuffers/%BUILD_TYPE%/",


Will %BUILD_TYPE% always be the final component of the directory path? If so, we could remove the %BUILD_TYPE% placeholder and just add it to the source path inside BuiltFile.

It's ok as-is for now, though.

It is, but it does not have access to the installer in the constructor of BuiltFile.
If I understand correctly, obtaining the build configuration requires access to installer.debug.

dbort · 2024-08-23T00:04:34Z

setup.py

-            src: The path to the file to install, relative to the cmake-out
-                directory. May be an fnmatch-style glob that matches exactly one
-                file.
+            src_path: The path to the file to install without name, relative to the cmake-out


Please mention %BUILD_TYPE% here and how to use it.

python3kgae · 2024-08-23T03:59:22Z

This is looking great!

Consider splitting this into some smaller PRs. E.g. we can probably merge the setup.py stuff pretty quickly, but the FileDataLoader might take longer. And, in general, we prefer to have smaller self-contained PRs.

Created 7 small PRs. Need more time for FileDataLoader.

python3kgae · 2024-11-19T14:49:51Z

Close after split to small PRs.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 13, 2024

cccclai requested review from dbort, tarun292 and jackzhxng August 13, 2024 03:20

manuelcandales assigned dbort Aug 13, 2024

manuelcandales added module: build/install Issues related to the cmake and buck2 builds, and to installing ExecuTorch triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Aug 13, 2024

python3kgae commented Aug 13, 2024

View reviewed changes

build/resolve_buck.py Show resolved Hide resolved

kirklandsign reviewed Aug 13, 2024

View reviewed changes

extension/data_loader/file_data_loader.cpp Outdated Show resolved Hide resolved

dbort requested changes Aug 14, 2024

View reviewed changes

python3kgae marked this pull request as draft August 14, 2024 20:39

python3kgae added 4 commits August 20, 2024 18:43

Fix lint

8a8b990

Move ssize_t to compiler.h.

d43591b

Update per comment. Switch to bat for Windows.

1b8899c

python3kgae force-pushed the win_bld branch from 37f4e3c to 1b8899c Compare August 21, 2024 02:11

python3kgae marked this pull request as ready for review August 21, 2024 02:15

dbort requested changes Aug 23, 2024

View reviewed changes

python3kgae closed this Nov 19, 2024

vortex-captain mentioned this pull request Nov 20, 2024

Windows x64 Build with support for xnnpack and llama example #6979

Closed

chinazhangchao mentioned this pull request Dec 6, 2024

Support build on native Windows x64 #7217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] fix build issues for native Windows. #4681

[Build] fix build issues for native Windows. #4681

python3kgae commented Aug 13, 2024

pytorch-bot bot commented Aug 13, 2024 •

edited

Loading

kirklandsign left a comment

python3kgae commented Aug 13, 2024

dbort left a comment

dbort Aug 14, 2024

dbort Aug 14, 2024

dbort Aug 14, 2024

python3kgae Aug 21, 2024

dbort Aug 14, 2024

python3kgae Aug 21, 2024

dbort Aug 14, 2024

python3kgae commented Aug 14, 2024

python3kgae commented Aug 21, 2024

dbort left a comment

dbort Aug 22, 2024

dbort Aug 22, 2024

dbort Aug 22, 2024

dbort Aug 22, 2024

dbort Aug 22, 2024

dbort Aug 23, 2024

dbort Aug 23, 2024

dbort Aug 23, 2024

python3kgae Aug 23, 2024

dbort Aug 23, 2024

python3kgae commented Aug 23, 2024

python3kgae commented Nov 19, 2024

		debug = os.environ.get("DEBUG", 0)
		cfg = "Debug" if debug else "Release"

	# avoid def flat.exe() on windows.
	# Remove .exe suffix on windows.

[Build] fix build issues for native Windows. #4681

[Build] fix build issues for native Windows. #4681

Conversation

python3kgae commented Aug 13, 2024

pytorch-bot bot commented Aug 13, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4681

❌ 7 New Failures

kirklandsign left a comment

Choose a reason for hiding this comment

python3kgae commented Aug 13, 2024

dbort left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

python3kgae commented Aug 14, 2024

python3kgae commented Aug 21, 2024

dbort left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

python3kgae commented Aug 23, 2024

python3kgae commented Nov 19, 2024

pytorch-bot bot commented Aug 13, 2024 •

edited

Loading