diff --git a/README.md b/README.md index 3ad91f3efc..33c7c60c4b 100644 --- a/README.md +++ b/README.md @@ -36,14 +36,16 @@ important concepts to help understanding the internals of the SDK: * **SDK**: A set of tools. For example, 'sdk-1.5.6-32bit' is an SDK consisting of the tools `clang-3.2-32bit`, `node-0.10.17-32bit`, `python-2.7.5.1-32bit` and `emscripten-1.5.6`. -* **Active Tool/SDK**: Emscripten SDK stores compiler configuration in a file - called `.emscripten` within the emsdk directory. This file points to paths - for Emscripten, Python, Clang and so on. If the configuration file points to a - Tool in a specific directory, then that tool is denoted as being - **active**. This mechanism allows switching between different installed - tools and SDKs. +* **Active Tool/SDK**: Emscripten stores compiler configuration in a + user-specific file **~/.emscripten**. This file points to paths for + Emscripten, Python, Clang and so on. If the file ~/.emscripten is configured + to point to a Tool in a specific directory, then that tool is denoted as being + **active**. The Emscripten Command Prompt always gives access to the currently + active Tools. This mechanism allows switching between different installed SDK + versions easily. * **emsdk**: This is the name of the manager script that Emscripten SDK is - accessed through. Most operations are of the form `emsdk `. + accessed through. Most operations are of the form `emsdk command`. To access + the emsdk script, launch the Emscripten Command Prompt. ## System Requirements @@ -134,11 +136,11 @@ using git, and compile the package on demand. When you run `emsdk list`, it will group the Tools and SDKs under these two categories. -To obtain and build latest wasm SDK from source, run +To obtain and build latest upstream wasm SDK from source, run ``` -emsdk install sdk-main-64bit -emsdk activate sdk-main-64bit +emsdk install sdk-upstream-main-64bit +emsdk activate sdk-upstream-main-64bit ``` You can use this target for example to bootstrap developing patches to LLVM, @@ -154,7 +156,7 @@ https://emscripten.org/docs/contributing/developers_guide.html?highlight=develop ### When working on git branches compiled from source, how do I update to a newer compiler version? Unlike tags and precompiled versions, a few of the SDK packages are based on -"moving" git branches and compiled from source (e.g. sdk-main, +"moving" git branches and compiled from source (e.g. sdk-upstream-main, sdk-main, emscripten-main, binaryen-main). Because of that, the compiled versions will eventually go out of date as new commits are introduced to the development branches. To update an old compiled installation of one of @@ -177,6 +179,18 @@ activate `. Activating a tool will set up `~/.emscripten` to point to that particular tool. On Windows, you can pass the option `--permanent` to the `activate` command to register the environment permanently for the current user. Use `--system` to do this for all users. +### How do I build multiple projects with different SDK versions in parallel? + +By default, Emscripten locates all configuration files in the home directory of +the user. This may be a problem if you need to simultaneously build with +multiple Emscripten compiler versions, since the user home directory can only be +configured to point to one compiler at a time. This can be overcome by +specifying the '--embedded' option as a parameter to 'emsdk activate', which +will signal emsdk to generate the compiler configuration files inside the emsdk +root directory instead of the user home directory. Use this option also when it +is desirable to run emsdk in a fully portable mode that does not touch any files +outside the emsdk directory. + ### How do I track the latest Emscripten development with the SDK? A common and supported use case of the Emscripten SDK is to enable the workflow @@ -184,11 +198,11 @@ where you directly interact with the github repositories. This allows you to obtain new features and latest fixes immediately as they are pushed to the github repository, without having to wait for release to be tagged. You do not need a github account or a fork of Emscripten to do this. To switch to using the -latest git development branch `main`, run the following: +latest upstream git development branch `main`, run the following: emsdk install git-1.9.4 # Install git. Skip if the system already has it. - emsdk install sdk-main-64bit # Clone+pull the latest emscripten-core/emscripten/main. - emsdk activate sdk-main-64bit # Set the main SDK as the currently active one. + emsdk install sdk-upstream-main-64bit # Clone+pull the latest emscripten-core/emscripten/main. + emsdk activate sdk-upstream-main-64bit # Set the main SDK as the currently active one. ### How do I use my own Emscripten github fork with the SDK? diff --git a/bazel/emscripten_deps.bzl b/bazel/emscripten_deps.bzl index 41db42b223..b1dae17b6a 100644 --- a/bazel/emscripten_deps.bzl +++ b/bazel/emscripten_deps.bzl @@ -14,7 +14,7 @@ filegroup( "emscripten/cache/sysroot/include/c++/v1/**", "emscripten/cache/sysroot/include/compat/**", "emscripten/cache/sysroot/include/**", - "lib/clang/17/include/**", + "lib/clang/16/include/**", ]), ) @@ -53,16 +53,12 @@ filegroup( srcs = [ "bin/clang{bin_extension}", "bin/llvm-ar{bin_extension}", - "bin/llvm-dwarfdump{bin_extension}", "bin/llvm-nm{bin_extension}", "bin/llvm-objcopy{bin_extension}", - "bin/wasm-ctor-eval{bin_extension}", "bin/wasm-emscripten-finalize{bin_extension}", "bin/wasm-ld{bin_extension}", - "bin/wasm-metadce{bin_extension}", "bin/wasm-opt{bin_extension}", - "bin/wasm-split{bin_extension}", - "bin/wasm2js{bin_extension}", + "bin/wasm-metadce{bin_extension}", ":emcc_common", ] + glob( include = [ diff --git a/bazel/emscripten_toolchain/toolchain.bzl b/bazel/emscripten_toolchain/toolchain.bzl index f1ea1c819e..2bddd56ccf 100644 --- a/bazel/emscripten_toolchain/toolchain.bzl +++ b/bazel/emscripten_toolchain/toolchain.bzl @@ -914,7 +914,7 @@ def _impl(ctx): "-iwithsysroot" + "/include/compat", "-iwithsysroot" + "/include", "-isystem", - emscripten_dir + "/lib/clang/17/include", + emscripten_dir + "/lib/clang/16/include", ], ), # Inputs and outputs @@ -1081,7 +1081,7 @@ def _impl(ctx): emscripten_dir + "/emscripten/cache/sysroot/include/c++/v1", emscripten_dir + "/emscripten/cache/sysroot/include/compat", emscripten_dir + "/emscripten/cache/sysroot/include", - emscripten_dir + "/lib/clang/17/include", + emscripten_dir + "/lib/clang/16/include", ] artifact_name_patterns = [] diff --git a/bazel/revisions.bzl b/bazel/revisions.bzl index 5ebed5077c..e55a49582b 100644 --- a/bazel/revisions.bzl +++ b/bazel/revisions.bzl @@ -2,34 +2,6 @@ # DO NOT MODIFY EMSCRIPTEN_TAGS = { - "3.1.34": struct( - hash = "2fdd6b9e5b67d5b62f84d0501a876513ff118ef1", - sha_linux = "dd3713f077072dcdb811f934d6685187daa47c424039e31cba83633c8d1681b1", - sha_mac = "3824609ee9b7c9919e29b19775d495a16778adb981867901f4bc503fe2f65d7d", - sha_mac_arm64 = "72728637171df46e7cd22f90537dd6faf1d4809ed1befc504ff96768c82f0e0f", - sha_win = "7538d1a1e0d586bd0723f595557551b05d724a5803132949a6fafb8b056af995", - ), - "3.1.33": struct( - hash = "49b960bd03b3a9da478a08541ce6eafe792a58a8", - sha_linux = "eab02b3f4b7c076974452ba602f908a36adf597afa15b16095b441f191ede1bb", - sha_mac = "b8dad3cddb19c1daf9dae99020bd17b903ae9649cfc58e433ea4951e758804de", - sha_mac_arm64 = "fbf03d06c7503f091191e440b8ea577d65b3261167cdb47359d053f12888974b", - sha_win = "031f951668eaeea39bd9363abb3f514efc3401506374984fa9b1d7ba3130a62f", - ), - "3.1.32": struct( - hash = "29ad1037cd6b99e5d8a1bd75bc188c1e9a6fda8d", - sha_linux = "25fa252e9fc674d1bcef35b3a10dd85024aa93c843b8067f8d917e5151968ffc", - sha_mac = "7881714e7738eb183b5a421bb2b907e96359e791ad0a622be6e7f5690a16b9d6", - sha_mac_arm64 = "04eede7352aca4b6fc1c111a8b31d00e8aa40547c3cd062ff9be4ffe1ed98d95", - sha_win = "22c3429eb1e6051bda46e9c02c14eca1ae3749ba8c411fbd5a3b51e3b9623161", - ), - "3.1.31": struct( - hash = "1eec24930cb2f56f6d9cd10ffcb031e27ea4157a", - sha_linux = "5952523c0c58cfc7c8839c1d3fe42ff34af5d8721231306ee432063dfacf96ca", - sha_mac = "13482cf3cb29f423f2037b9dc2b9e4ff72d0a49fcd471bbaa9b76d9f86f31d82", - sha_mac_arm64 = "654a35af16be5eeb2082e68fb36190fe76de28fa2da75ac0d2197482a203f39a", - sha_win = "493c29f5a505ccd9687036ee4c580d190b1c32b286be0e751a78e68997cec8b2", - ), "3.1.30": struct( hash = "dc1fdcfd3f5b9d29cb1ebdf15e6e845bef9b0cc1", sha_linux = "151d7afdfb728e1e55ed1d100e4d3fbd20925fd65f3c3b9e093061a2c89dcac7", diff --git a/emscripten-releases-tags.json b/emscripten-releases-tags.json index 52076a99d9..be06c022f5 100644 --- a/emscripten-releases-tags.json +++ b/emscripten-releases-tags.json @@ -1,6 +1,6 @@ { "aliases": { - "latest": "3.1.34", + "latest": "3.1.30", "latest-sdk": "latest", "latest-arm64-linux": "3.1.21", "latest-64bit": "latest", @@ -10,14 +10,6 @@ "latest-releases-upstream": "latest" }, "releases": { - "3.1.34": "2fdd6b9e5b67d5b62f84d0501a876513ff118ef1", - "3.1.34-asserts": "385382932c18a1312fff88000c4f83c2b9d1bb44", - "3.1.33": "49b960bd03b3a9da478a08541ce6eafe792a58a8", - "3.1.33-asserts": "e3ca2c6756b75cf6c6daa40276de0f25218e04a7", - "3.1.32": "29ad1037cd6b99e5d8a1bd75bc188c1e9a6fda8d", - "3.1.32-asserts": "2811c849256ec5b62b4ec32fb8369e5f3c9a54b1", - "3.1.31": "1eec24930cb2f56f6d9cd10ffcb031e27ea4157a", - "3.1.31-asserts": "48488847a38bb9cfb36e7397bea21ab2bb062680", "3.1.30": "dc1fdcfd3f5b9d29cb1ebdf15e6e845bef9b0cc1", "3.1.30-asserts": "21cca44e843267533c3d0b258b46c37bd142a2d7", "3.1.29": "d949f1b99a477d4b0b54d95413df3688afa69d0a", diff --git a/emsdk.py b/emsdk.py index 6788120d32..b789ea5297 100644 --- a/emsdk.py +++ b/emsdk.py @@ -256,11 +256,14 @@ def vswhere(version): if not program_files: program_files = os.environ['ProgramFiles'] vswhere_path = os.path.join(program_files, 'Microsoft Visual Studio', 'Installer', 'vswhere.exe') - # Source: https://learn.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2022 - tools_arch = 'ARM64' if ARCH == 'aarch64' else 'x86.x64' - # The "-products *" allows detection of Build Tools, the "-prerelease" allows detection of Preview version - # of Visual Studio and Build Tools. - output = json.loads(subprocess.check_output([vswhere_path, '-latest', '-products', '*', '-prerelease', '-version', '[%s.0,%s.0)' % (version, version + 1), '-requires', 'Microsoft.VisualStudio.Component.VC.Tools.' + tools_arch, '-property', 'installationPath', '-format', 'json'])) + output = json.loads(subprocess.check_output([vswhere_path, '-latest', '-version', '[%s.0,%s.0)' % (version, version + 1), '-requires', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', '-property', 'installationPath', '-format', 'json'])) + # Visual Studio 2017 Express is not included in the above search, and it + # does not have the VC.Tools.x86.x64 tool, so do a catch-all attempt as a + # fallback, to detect Express version. + if not output: + output = json.loads(subprocess.check_output([vswhere_path, '-latest', '-version', '[%s.0,%s.0)' % (version, version + 1), '-products', '*', '-property', 'installationPath', '-format', 'json'])) + if not output: + return '' return str(output[0]['installationPath']) except Exception: return '' @@ -281,30 +284,36 @@ def vs_filewhere(installation_path, platform, file): # Detect which CMake generator to use when building on Windows if '--mingw' in sys.argv: CMAKE_GENERATOR = 'MinGW Makefiles' - elif '--vs2022' in sys.argv: - CMAKE_GENERATOR = 'Visual Studio 17' + elif '--vs2017' in sys.argv: + CMAKE_GENERATOR = 'Visual Studio 15' elif '--vs2019' in sys.argv: CMAKE_GENERATOR = 'Visual Studio 16' - elif len(vswhere(17)) > 0: - CMAKE_GENERATOR = 'Visual Studio 17' - elif len(vswhere(16)) > 0: - CMAKE_GENERATOR = 'Visual Studio 16' - elif which('mingw32-make') is not None and which('g++') is not None: - CMAKE_GENERATOR = 'MinGW Makefiles' else: - # No detected generator - CMAKE_GENERATOR = '' + vs2019_exists = len(vswhere(16)) > 0 + vs2017_exists = len(vswhere(15)) > 0 + mingw_exists = which('mingw32-make') is not None and which('g++') is not None + if vs2019_exists: + CMAKE_GENERATOR = 'Visual Studio 16' + elif vs2017_exists: + # VS2017 has an LLVM build issue, see + # https://github.com/kripken/emscripten-fastcomp/issues/185 + CMAKE_GENERATOR = 'Visual Studio 15' + elif mingw_exists: + CMAKE_GENERATOR = 'MinGW Makefiles' + else: + # No detected generator + CMAKE_GENERATOR = '' -sys.argv = [a for a in sys.argv if a not in ('--mingw', '--vs2019', '--vs2022')] +sys.argv = [a for a in sys.argv if a not in ('--mingw', '--vs2017', '--vs2019')] # Computes a suitable path prefix to use when building with a given generator. def cmake_generator_prefix(): - if CMAKE_GENERATOR == 'Visual Studio 17': - return '_vs2022' if CMAKE_GENERATOR == 'Visual Studio 16': return '_vs2019' + if CMAKE_GENERATOR == 'Visual Studio 15': + return '_vs2017' elif CMAKE_GENERATOR == 'MinGW Makefiles': return '_mingw' # Unix Makefiles do not specify a path prefix for backwards path compatibility @@ -852,7 +861,14 @@ def decide_cmake_build_type(tool): # The root directory of the build. def llvm_build_dir(tool): - generator_suffix = cmake_generator_prefix() + generator_suffix = '' + if CMAKE_GENERATOR == 'Visual Studio 15': + generator_suffix = '_vs2017' + elif CMAKE_GENERATOR == 'Visual Studio 16': + generator_suffix = '_vs2019' + elif CMAKE_GENERATOR == 'MinGW Makefiles': + generator_suffix = '_mingw' + bitness_suffix = '_32' if tool.bitness == 32 else '_64' if hasattr(tool, 'git_branch'): @@ -900,38 +916,112 @@ def build_env(generator): # See https://groups.google.com/forum/#!topic/emscripten-discuss/5Or6QIzkqf0 if MACOS: build_env['CXXFLAGS'] = ((build_env['CXXFLAGS'] + ' ') if hasattr(build_env, 'CXXFLAGS') else '') + '-stdlib=libc++' - if WINDOWS: - # MSBuild.exe has an internal mechanism to avoid N^2 oversubscription of threads in its two-tier build model, see - # https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/ - build_env['UseMultiToolTask'] = 'true' - build_env['EnforceProcessCountAcrossBuilds'] = 'true' + elif 'Visual Studio 15' in generator or 'Visual Studio 16' in generator: + if 'Visual Studio 16' in generator: + path = vswhere(16) + else: + path = vswhere(15) + + # Configuring CMake for Visual Studio needs and env. var VCTargetsPath to be present. + # How this is supposed to work is unfortunately very undocumented. See + # https://discourse.cmake.org/t/cmake-failed-to-get-the-value-of-vctargetspath-with-vs2019-16-7/1839/16 + # for some conversation. Try a couple of common paths if one of them would work. + # In the future as new versions of VS come out, we likely need to add new paths into this list. + if 'VCTargetsPath' not in build_env: + vctargets_paths = [ + os.path.join(path, 'MSBuild\\Microsoft\\VC\\v160\\'), + os.path.join(path, 'Common7\\IDE\\VC\\VCTargets') + ] + for p in vctargets_paths: + if os.path.isfile(os.path.join(p, 'Microsoft.Cpp.Default.props')): + debug_print('Set env. var VCTargetsPath=' + p + ' for CMake.') + build_env['VCTargetsPath'] = p + break + else: + debug_print('Searched path ' + p + ' as candidate for VCTargetsPath, not working.') + + if 'VCTargetsPath' not in build_env: + errlog('Unable to locate Visual Studio compiler installation for generator "' + generator + '"!') + errlog('Either rerun installation in Visual Studio Command Prompt, or locate directory to Microsoft.Cpp.Default.props manually') + sys.exit(1) + + # CMake and VS2017 cl.exe needs to have mspdb140.dll et al. in its PATH. + vc_bin_paths = [vs_filewhere(path, 'amd64', 'cl.exe'), + vs_filewhere(path, 'x86', 'cl.exe')] + for path in vc_bin_paths: + if os.path.isdir(path): + build_env['PATH'] = build_env['PATH'] + ';' + path + return build_env -def make_build(build_root, build_type): - debug_print('make_build(build_root=' + build_root + ', build_type=' + build_type + ')') +def get_generator_for_sln_file(sln_file): + contents = open(sln_file, 'r').read() + if '# Visual Studio 16' in contents or '# Visual Studio Version 16' in contents: # VS2019 + return 'Visual Studio 16' + if '# Visual Studio 15' in contents: # VS2017 + return 'Visual Studio 15' + raise Exception('Unknown generator used to build solution file ' + sln_file) + + +def find_msbuild(sln_file): + # The following logic attempts to find a Visual Studio version specific + # MSBuild.exe from a list of known locations. + generator = get_generator_for_sln_file(sln_file) + debug_print('find_msbuild looking for generator ' + str(generator)) + if generator == 'Visual Studio 16': # VS2019 + path = vswhere(16) + search_paths = [os.path.join(path, 'MSBuild/Current/Bin'), + os.path.join(path, 'MSBuild/15.0/Bin/amd64'), + os.path.join(path, 'MSBuild/15.0/Bin')] + elif generator == 'Visual Studio 15': # VS2017 + path = vswhere(15) + search_paths = [os.path.join(path, 'MSBuild/15.0/Bin/amd64'), + os.path.join(path, 'MSBuild/15.0/Bin')] + else: + raise Exception('Unknown generator!') + + for path in search_paths: + p = os.path.join(path, 'MSBuild.exe') + debug_print('Searching for MSBuild.exe: ' + p) + if os.path.isfile(p): + return p + debug_print('MSBuild.exe in PATH? ' + str(which('MSBuild.exe'))) + # Last fallback, try any MSBuild from PATH (might not be compatible, but best effort) + return which('MSBuild.exe') + + +def make_build(build_root, build_type, build_target_platform='x64'): + debug_print('make_build(build_root=' + build_root + ', build_type=' + build_type + ', build_target_platform=' + build_target_platform + ')') if CPU_CORES > 1: print('Performing a parallel build with ' + str(CPU_CORES) + ' cores.') else: print('Performing a singlethreaded build.') - make = ['cmake', '--build', '.', '--config', build_type] - if 'Visual Studio' in CMAKE_GENERATOR: - # Visual Studio historically has had a two-tier problem in its build system design. A single MSBuild.exe instance only governs - # the build of a single project (.exe/.lib/.dll) in a solution. Passing the -j parameter above will only enable multiple MSBuild.exe - # instances to be spawned to build multiple projects in parallel, but each MSBuild.exe is still singlethreaded. - # To enable each MSBuild.exe instance to also compile several .cpp files in parallel inside a single project, pass the extra - # MSBuild.exe specific "Multi-ToolTask" (MTT) setting /p:CL_MPCount. This enables each MSBuild.exe to parallelize builds wide. - # This requires CMake 3.12 or newer. - make += ['-j', str(CPU_CORES), '--', '/p:CL_MPCount=' + str(CPU_CORES)] + generator_to_use = CMAKE_GENERATOR + + if WINDOWS: + if 'Visual Studio' in CMAKE_GENERATOR: + solution_name = str(subprocess.check_output(['dir', '/b', '*.sln'], shell=True, cwd=build_root).decode('utf-8').strip()) + generator_to_use = get_generator_for_sln_file(os.path.join(build_root, solution_name)) + # Disabled for now: Don't pass /maxcpucount argument to msbuild, since it + # looks like when building, msbuild already automatically spawns the full + # amount of logical cores the system has, and passing the number of + # logical cores here has been observed to give a quadratic N*N explosion + # on the number of spawned processes (e.g. on a Core i7 5960X with 16 + # logical cores, it would spawn 16*16=256 cl.exe processes, which would + # start crashing when running out of system memory) + # make = [find_msbuild(os.path.join(build_root, solution_name)), '/maxcpucount:' + str(CPU_CORES), '/t:Build', '/p:Configuration=' + build_type, '/nologo', '/verbosity:minimal', solution_name] + make = [find_msbuild(os.path.join(build_root, solution_name)), '/t:Build', '/p:Configuration=' + build_type, '/p:Platform=' + build_target_platform, '/nologo', '/verbosity:minimal', solution_name] + else: + make = ['mingw32-make', '-j' + str(CPU_CORES)] else: - # Pass -j to native make, CMake might not support -j option. - make += ['--', '-j', str(CPU_CORES)] + make = ['cmake', '--build', '.', '--', '-j' + str(CPU_CORES)] # Build try: print('Running build: ' + str(make)) - ret = subprocess.check_call(make, cwd=build_root, env=build_env(CMAKE_GENERATOR)) + ret = subprocess.check_call(make, cwd=build_root, env=build_env(generator_to_use)) if ret != 0: errlog('Build failed with exit code ' + ret + '!') errlog('Working directory: ' + build_root) @@ -1018,47 +1108,6 @@ def xcode_sdk_version(): return subprocess.checkplatform.mac_ver()[0].split('.') -def cmake_target_platform(tool): - # Source: https://cmake.org/cmake/help/latest/generator/Visual%20Studio%2017%202022.html#platform-selection - if hasattr(tool, 'arch'): - if tool.arch == 'aarch64': - return 'ARM64' - elif tool.arch == 'x86_64': - return 'x64' - elif tool.arch == 'x86': - return 'Win32' - if ARCH == 'aarch64': - return 'ARM64' - else: - return 'x64' if tool.bitness == 64 else 'Win32' - - -def cmake_host_platform(): - # Source: https://cmake.org/cmake/help/latest/generator/Visual%20Studio%2017%202022.html#toolset-selection - arch_to_cmake_host_platform = { - 'aarch64': 'ARM64', - 'arm': 'ARM', - 'x86_64': 'x64', - 'x86': 'x86' - } - return arch_to_cmake_host_platform[ARCH] - - -def get_generator_and_config_args(tool): - args = [] - cmake_generator = CMAKE_GENERATOR - if 'Visual Studio 16' in CMAKE_GENERATOR or 'Visual Studio 17' in CMAKE_GENERATOR: # VS2019 or VS2022 - # With Visual Studio 16 2019, CMake changed the way they specify target arch. - # Instead of appending it into the CMake generator line, it is specified - # with a -A arch parameter. - args += ['-A', cmake_target_platform(tool)] - args += ['-Thost=' + cmake_host_platform()] - elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: - cmake_generator += ' Win64' - args += ['-Thost=x64'] - return (cmake_generator, args) - - def build_llvm(tool): debug_print('build_llvm(' + str(tool) + ')') llvm_root = tool.installation_path() @@ -1085,16 +1134,15 @@ def build_llvm(tool): targets_to_build = 'WebAssembly;AArch64' else: targets_to_build = 'WebAssembly' - cmake_generator, args = get_generator_and_config_args(tool) - args += ['-DLLVM_TARGETS_TO_BUILD=' + targets_to_build, - '-DLLVM_INCLUDE_EXAMPLES=OFF', - '-DLLVM_INCLUDE_TESTS=' + tests_arg, - '-DCLANG_INCLUDE_TESTS=' + tests_arg, - '-DLLVM_ENABLE_ASSERTIONS=' + ('ON' if enable_assertions else 'OFF'), - # Disable optional LLVM dependencies, these can cause unwanted .so dependencies - # that prevent distributing the generated compiler for end users. - '-DLLVM_ENABLE_LIBXML2=OFF', '-DLLVM_ENABLE_TERMINFO=OFF', '-DLLDB_ENABLE_LIBEDIT=OFF', - '-DLLVM_ENABLE_LIBEDIT=OFF', '-DLLVM_ENABLE_LIBPFM=OFF'] + args = ['-DLLVM_TARGETS_TO_BUILD=' + targets_to_build, + '-DLLVM_INCLUDE_EXAMPLES=OFF', + '-DLLVM_INCLUDE_TESTS=' + tests_arg, + '-DCLANG_INCLUDE_TESTS=' + tests_arg, + '-DLLVM_ENABLE_ASSERTIONS=' + ('ON' if enable_assertions else 'OFF'), + # Disable optional LLVM dependencies, these can cause unwanted .so dependencies + # that prevent distributing the generated compiler for end users. + '-DLLVM_ENABLE_LIBXML2=OFF', '-DLLVM_ENABLE_TERMINFO=OFF', '-DLLDB_ENABLE_LIBEDIT=OFF', + '-DLLVM_ENABLE_LIBEDIT=OFF', '-DLLVM_ENABLE_LIBPFM=OFF'] # LLVM build system bug: compiler-rt does not build on Windows. It insists on performing a CMake install step that writes to C:\Program Files. Attempting # to reroute that to build_root directory then fails on an error # file INSTALL cannot find @@ -1102,6 +1150,16 @@ def build_llvm(tool): # (there instead of $(Configuration), one would need ${CMAKE_BUILD_TYPE} ?) # It looks like compiler-rt is not compatible to build on Windows? args += ['-DLLVM_ENABLE_PROJECTS=clang;lld'] + cmake_generator = CMAKE_GENERATOR + if 'Visual Studio 16' in CMAKE_GENERATOR: # VS2019 + # With Visual Studio 16 2019, CMake changed the way they specify target arch. + # Instead of appending it into the CMake generator line, it is specified + # with a -A arch parameter. + args += ['-A', 'x64' if tool.bitness == 64 else 'x86'] + args += ['-Thost=x64'] + elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: + cmake_generator += ' Win64' + args += ['-Thost=x64'] if os.getenv('LLVM_CMAKE_ARGS'): extra_args = os.environ['LLVM_CMAKE_ARGS'].split(',') @@ -1114,7 +1172,7 @@ def build_llvm(tool): return False # Make - success = make_build(build_root, build_type) + success = make_build(build_root, build_type, 'x64' if tool.bitness == 64 else 'Win32') return success @@ -1132,7 +1190,17 @@ def build_ninja(tool): build_type = decide_cmake_build_type(tool) # Configure - cmake_generator, args = get_generator_and_config_args(tool) + cmake_generator = CMAKE_GENERATOR + args = [] + if 'Visual Studio 16' in CMAKE_GENERATOR: # VS2019 + # With Visual Studio 16 2019, CMake changed the way they specify target arch. + # Instead of appending it into the CMake generator line, it is specified + # with a -A arch parameter. + args += ['-A', 'x64' if tool.bitness == 64 else 'x86'] + args += ['-Thost=x64'] + elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: + cmake_generator += ' Win64' + args += ['-Thost=x64'] cmakelists_dir = os.path.join(src_root) success = cmake_configure(cmake_generator, build_root, cmakelists_dir, build_type, args) @@ -1140,7 +1208,7 @@ def build_ninja(tool): return False # Make - success = make_build(build_root, build_type) + success = make_build(build_root, build_type, 'x64' if tool.bitness == 64 else 'Win32') if success: bin_dir = os.path.join(root, 'bin') @@ -1171,8 +1239,17 @@ def build_ccache(tool): build_type = decide_cmake_build_type(tool) # Configure - cmake_generator, args = get_generator_and_config_args(tool) - args += ['-DZSTD_FROM_INTERNET=ON'] + cmake_generator = CMAKE_GENERATOR + args = ['-DZSTD_FROM_INTERNET=ON'] + if 'Visual Studio 16' in CMAKE_GENERATOR: # VS2019 + # With Visual Studio 16 2019, CMake changed the way they specify target arch. + # Instead of appending it into the CMake generator line, it is specified + # with a -A arch parameter. + args += ['-A', 'x64' if tool.bitness == 64 else 'x86'] + args += ['-Thost=x64'] + elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: + cmake_generator += ' Win64' + args += ['-Thost=x64'] cmakelists_dir = os.path.join(src_root) success = cmake_configure(cmake_generator, build_root, cmakelists_dir, build_type, args) @@ -1180,7 +1257,7 @@ def build_ccache(tool): return False # Make - success = make_build(build_root, build_type) + success = make_build(build_root, build_type, 'x64' if tool.bitness == 64 else 'Win32') if success: bin_dir = os.path.join(root, 'bin') @@ -1332,15 +1409,24 @@ def emscripten_post_install(tool): build_root = optimizer_build_root(tool) build_type = decide_cmake_build_type(tool) + args = [] + # Configure - cmake_generator, args = get_generator_and_config_args(tool) + cmake_generator = CMAKE_GENERATOR + if 'Visual Studio 16' in CMAKE_GENERATOR: # VS2019 + # With Visual Studio 16 2019, CMake changed the way they specify target arch. + # Instead of appending it into the CMake generator line, it is specified + # with a -A arch parameter. + args += ['-A', 'x64' if tool.bitness == 64 else 'x86'] + elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: + cmake_generator += ' Win64' success = cmake_configure(cmake_generator, build_root, src_root, build_type, args) if not success: return False # Make - success = make_build(build_root, build_type) + success = make_build(build_root, build_type, 'x64' if tool.bitness == 64 else 'Win32') if not success: return False @@ -1378,8 +1464,16 @@ def build_binaryen_tool(tool): build_type = decide_cmake_build_type(tool) # Configure - cmake_generator, args = get_generator_and_config_args(tool) - args += ['-DENABLE_WERROR=0'] # -Werror is not useful for end users + args = ['-DENABLE_WERROR=0'] # -Werror is not useful for end users + + cmake_generator = CMAKE_GENERATOR + if 'Visual Studio 16' in CMAKE_GENERATOR: # VS2019 + # With Visual Studio 16 2019, CMake changed the way they specify target arch. + # Instead of appending it into the CMake generator line, it is specified + # with a -A arch parameter. + args += ['-A', 'x64' if tool.bitness == 64 else 'x86'] + elif 'Visual Studio' in CMAKE_GENERATOR and tool.bitness == 64: + cmake_generator += ' Win64' if 'Visual Studio' in CMAKE_GENERATOR: if BUILD_FOR_TESTING: @@ -1390,7 +1484,7 @@ def build_binaryen_tool(tool): return False # Make - success = make_build(build_root, build_type) + success = make_build(build_root, build_type, 'x64' if tool.bitness == 64 else 'Win32') # Deploy scripts needed from source repository to build directory remove_tree(os.path.join(build_root, 'scripts')) @@ -1436,8 +1530,6 @@ def get_required_path(active_tools): path_add = [to_native_path(EMSDK_PATH)] for tool in active_tools: if hasattr(tool, 'activated_path'): - if hasattr(tool, 'activated_path_skip') and which(tool.activated_path_skip): - continue path = to_native_path(tool.expand_vars(tool.activated_path)) path_add.append(path) return path_add @@ -1863,10 +1955,6 @@ def install_tool(self): elif hasattr(self, 'git_branch'): success = git_clone_checkout_and_pull(url, self.installation_path(), self.git_branch) elif url.endswith(ARCHIVE_SUFFIXES): - global ARCH - if WINDOWS and ARCH == 'aarch64': - errlog('No support for Windows on Arm, fallback to x64') - ARCH = 'x86_64' success = download_and_unzip(url, self.installation_path(), filename_prefix=getattr(self, 'zipfile_prefix', '')) else: assert False, 'unhandled url type: ' + url @@ -2023,15 +2111,15 @@ def resolve_sdk_aliases(name, verbose=False): return name -def find_latest_sdk(): - return 'sdk-releases-%s-64bit' % (find_latest_hash()) +def find_latest_sdk(which): + return 'sdk-releases-%s-%s-64bit' % (which, find_latest_hash()) def find_tot_sdk(): debug_print('Fetching emscripten-releases repository...') global extra_release_tag extra_release_tag = get_emscripten_releases_tot() - return 'sdk-releases-%s-64bit' % (extra_release_tag) + return 'sdk-releases-upstream-%s-64bit' % (extra_release_tag) def parse_emscripten_version(emscripten_root): @@ -2161,7 +2249,7 @@ def get_installed_sdk_version(): return None with open(version_file) as f: version = f.read() - return version.split('-')[1] + return version.split('-')[2] # Get a list of tags for emscripten-releases. @@ -2598,8 +2686,8 @@ def error_on_missing_tool(name): def expand_sdk_name(name, activating): if 'upstream-master' in name: - errlog('upstream-master SDK has been renamed main') - name = name.replace('upstream-master', 'main') + errlog('upstream-master SDK has been renamed upstream-main') + name = name.replace('upstream-master', 'upstream-main') if 'fastcomp' in name: exit_with_error('the fastcomp backend is no longer supported. Please use an older version of emsdk (for example 3.1.29) if you want to install the old fastcomp-based SDK') if name in ('tot', 'sdk-tot', 'tot-upstream'): @@ -2611,34 +2699,34 @@ def expand_sdk_name(name, activating): installed = get_installed_sdk_version() if installed: debug_print('activating currently installed SDK; not updating tot version') - return 'sdk-releases-%s-64bit' % installed - return find_tot_sdk() - - if '-upstream' in name: - name = name.replace('-upstream', '') + return 'sdk-releases-upstream-%s-64bit' % installed + return str(find_tot_sdk()) name = resolve_sdk_aliases(name, verbose=True) # check if it's a release handled by an emscripten-releases version, # and if so use that by using the right hash. we support a few notations, - # x.y.z - # sdk-x.y.z-64bit + # x.y.z[-upstream] + # sdk-x.y.z[-upstream]-64bit # TODO: support short notation for old builds too? + backend = 'upstream' fullname = name + if '-upstream' in fullname: + fullname = name.replace('-upstream', '') version = fullname.replace('sdk-', '').replace('releases-', '').replace('-64bit', '').replace('tag-', '') sdk = 'sdk-' if not name.startswith('releases-') else '' releases_info = load_releases_info()['releases'] release_hash = get_release_hash(version, releases_info) if release_hash: # Known release hash - full_name = '%sreleases-%s-64bit' % (sdk, release_hash) + full_name = '%sreleases-%s-%s-64bit' % (sdk, backend, release_hash) print("Resolving SDK version '%s' to '%s'" % (version, full_name)) return full_name if len(version) == 40: global extra_release_tag extra_release_tag = version - return '%sreleases-%s-64bit' % (sdk, version) + return '%sreleases-%s-%s-64bit' % (sdk, backend, version) return name @@ -2706,7 +2794,7 @@ def main(args): purposes. Default: Enabled --disable-assertions: Forces assertions off during the build. - --vs2019/--vs2022: If building from source, overrides to build + --vs2017/--vs2019: If building from source, overrides to build using the specified compiler. When installing precompiled packages, this has no effect. Note: The same compiler specifier must be @@ -2729,7 +2817,7 @@ def main(args): if WINDOWS: print(''' - emsdk activate [--permanent] [--system] [--build=type] [--vs2019/--vs2022] + emsdk activate [--permanent] [--system] [--build=type] [--vs2017/--vs2019] - Activates the given tool or SDK in the environment of the current shell. @@ -2743,8 +2831,8 @@ def main(args): (uses Machine environment variables). - If a custom compiler version was used to override - the compiler to use, pass the same --vs2019/--vs2022 - parameter here to choose which version to activate. + the compiler to use, pass the same --vs2017/--vs2019 parameter + here to choose which version to activate. emcmdprompt.bat - Spawns a new command prompt window with the Emscripten environment active.''') @@ -2873,7 +2961,7 @@ def installed_sdk_text(name): print(' latest') print('') print('This is equivalent to installing/activating:') - print(' %s %s' % (find_latest_version(), installed_sdk_text(find_latest_sdk()))) + print(' %s %s' % (find_latest_version(), installed_sdk_text(find_latest_sdk('upstream')))) print('') else: print('Warning: your platform does not have precompiled SDKs available.') @@ -2881,10 +2969,14 @@ def installed_sdk_text(name): print('') print('All recent (non-legacy) installable versions are:') - releases_versions = sorted(load_releases_versions(), key=version_key, reverse=True) + releases_versions = sorted( + load_releases_versions(), + key=lambda x: [int(v) if v.isdigit() else -1 for v in x.split('.')], + reverse=True, + ) releases_info = load_releases_info()['releases'] for ver in releases_versions: - print(' %s %s' % (ver, installed_sdk_text('sdk-releases-%s-64bit' % get_release_hash(ver, releases_info)))) + print(' %s %s' % (ver, installed_sdk_text('sdk-releases-upstream-%s-64bit' % get_release_hash(ver, releases_info)))) print() # Use array to work around the lack of being able to mutate from enclosing diff --git a/emsdk_manifest.json b/emsdk_manifest.json index b14385b655..6649fb5cc4 100644 --- a/emsdk_manifest.json +++ b/emsdk_manifest.json @@ -31,7 +31,7 @@ { "id": "releases", - "version": "%releases-tag%", + "version": "upstream-%releases-tag%", "bitness": 64, "arch": "x86_64", "linux_url": "https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/%releases-tag%/wasm-binaries.tbz2", @@ -45,7 +45,7 @@ }, { "id": "releases", - "version": "%releases-tag%", + "version": "upstream-%releases-tag%", "bitness": 64, "arch": "aarch64", "macos_url": "https://storage.googleapis.com/webassembly/emscripten-releases-builds/mac/%releases-tag%/wasm-binaries-arm64.tbz2", @@ -204,7 +204,6 @@ "arch": "x86", "windows_url": "node-v14.15.5-win-x86.zip", "activated_path": "%installation_dir%/bin", - "activated_path_skip": "node", "activated_cfg": "NODE_JS='%installation_dir%/bin/node%.exe%'", "activated_env": "EMSDK_NODE=%installation_dir%/bin/node%.exe%" }, @@ -215,7 +214,6 @@ "bitness": 32, "linux_url": "node-v14.15.5-linux-armv7l.tar.xz", "activated_path": "%installation_dir%/bin", - "activated_path_skip": "node", "activated_cfg": "NODE_JS='%installation_dir%/bin/node%.exe%'", "activated_env": "EMSDK_NODE=%installation_dir%/bin/node%.exe%" }, @@ -228,7 +226,6 @@ "windows_url": "node-v14.15.5-win-x64.zip", "linux_url": "node-v14.15.5-linux-x64.tar.xz", "activated_path": "%installation_dir%/bin", - "activated_path_skip": "node", "activated_cfg": "NODE_JS='%installation_dir%/bin/node%.exe%'", "activated_env": "EMSDK_NODE=%installation_dir%/bin/node%.exe%" }, @@ -240,7 +237,6 @@ "macos_url": "node-v14.15.5-darwin-x64.tar.gz", "linux_url": "node-v14.15.5-linux-arm64.tar.xz", "activated_path": "%installation_dir%/bin", - "activated_path_skip": "node", "activated_cfg": "NODE_JS='%installation_dir%/bin/node%.exe%'", "activated_env": "EMSDK_NODE=%installation_dir%/bin/node%.exe%" }, @@ -547,56 +543,56 @@ "sdks": [ { - "version": "main", + "version": "upstream-main", "bitness": 64, "uses": ["python-3.9.2-nuget-64bit", "llvm-git-main-64bit", "node-14.18.2-64bit", "emscripten-main-64bit", "binaryen-main-64bit"], "os": "win" }, { - "version": "main", + "version": "upstream-main", "bitness": 64, "uses": ["python-3.9.2-64bit", "llvm-git-main-64bit", "node-14.18.2-64bit", "emscripten-main-64bit", "binaryen-main-64bit"], "os": "macos" }, { - "version": "main", + "version": "upstream-main", "bitness": 64, "uses": ["llvm-git-main-64bit", "node-14.18.2-64bit", "emscripten-main-64bit", "binaryen-main-64bit"], "os": "linux" }, { - "version": "main", + "version": "upstream-main", "bitness": 32, "uses": ["llvm-git-main-32bit", "emscripten-main-32bit", "binaryen-main-32bit"], "os": "linux" }, { - "version": "releases-%releases-tag%", + "version": "releases-upstream-%releases-tag%", "bitness": 64, - "uses": ["node-14.18.2-64bit", "releases-%releases-tag%-64bit"], + "uses": ["node-14.18.2-64bit", "releases-upstream-%releases-tag%-64bit"], "os": "linux", "custom_install_script": "emscripten_npm_install" }, { - "version": "releases-%releases-tag%", + "version": "releases-upstream-%releases-tag%", "bitness": 64, - "uses": ["node-14.18.2-64bit", "python-3.9.2-64bit", "releases-%releases-tag%-64bit"], + "uses": ["node-14.18.2-64bit", "python-3.9.2-64bit", "releases-upstream-%releases-tag%-64bit"], "os": "macos", "arch": "x86_64", "custom_install_script": "emscripten_npm_install" }, { - "version": "releases-%releases-tag%", + "version": "releases-upstream-%releases-tag%", "bitness": 64, - "uses": ["node-14.18.2-64bit", "python-3.9.2-64bit", "releases-%releases-tag%-64bit"], + "uses": ["node-14.18.2-64bit", "python-3.9.2-64bit", "releases-upstream-%releases-tag%-64bit"], "os": "macos", "arch": "aarch64", "custom_install_script": "emscripten_npm_install" }, { - "version": "releases-%releases-tag%", + "version": "releases-upstream-%releases-tag%", "bitness": 64, - "uses": ["node-14.18.2-64bit", "python-3.9.2-nuget-64bit", "java-8.152-64bit", "releases-%releases-tag%-64bit"], + "uses": ["node-14.18.2-64bit", "python-3.9.2-nuget-64bit", "java-8.152-64bit", "releases-upstream-%releases-tag%-64bit"], "os": "win", "custom_install_script": "emscripten_npm_install" } diff --git a/eng/Versions.props b/eng/Versions.props index 890884644a..89b94578a6 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -6,7 +6,7 @@ 8.0.0 preview 4 - 3.1.34 + 3.1.30 2.0.23 3.1.12 7.0.3 diff --git a/eng/clang-wrapper.sh b/eng/clang-wrapper.sh index df62999dcc..4a023e3591 100755 --- a/eng/clang-wrapper.sh +++ b/eng/clang-wrapper.sh @@ -2,12 +2,12 @@ # # This abomination of a script is meant to replace the symlinks -# pointing to the clang-17 binary and set the appropriate flags -# to avoid duplicating the 90MB clang-17 binary inside the nuget +# pointing to the clang-16 binary and set the appropriate flags +# to avoid duplicating the 90MB clang-16 binary inside the nuget # package. # CLANG_NAME=$(basename "$0") -CLANG_CC=$(dirname $0)/clang-17 +CLANG_CC=$(dirname $0)/clang-16 EXTRA_ARGS="" case $CLANG_NAME in diff --git a/eng/common/native/init-compiler.sh b/eng/common/native/init-compiler.sh index 9245bd1abc..7aee4213e1 100644 --- a/eng/common/native/init-compiler.sh +++ b/eng/common/native/init-compiler.sh @@ -63,7 +63,7 @@ if [ -z "$CLR_CC" ]; then # Set default versions if [ -z "$majorVersion" ]; then # note: gcc (all versions) and clang versions higher than 6 do not have minor version in file name, if it is zero. - if [ "$compiler" = "clang" ]; then versions="17 16 15 14 13 12 11 10 9 8 7 6.0 5.0 4.0 3.9 3.8 3.7 3.6 3.5" + if [ "$compiler" = "clang" ]; then versions="16 15 14 13 12 11 10 9 8 7 6.0 5.0 4.0 3.9 3.8 3.7 3.6 3.5" elif [ "$compiler" = "gcc" ]; then versions="12 11 10 9 8 7 6 5 4.9"; fi for version in $versions; do diff --git a/eng/emsdk.proj b/eng/emsdk.proj index 18de4d0a59..ced78c609e 100644 --- a/eng/emsdk.proj +++ b/eng/emsdk.proj @@ -43,9 +43,9 @@ - + + Condition="!Exists('$(UpstreamDirectory)\bin\clang-16') and !$([MSBuild]::IsOSPlatform(Windows))" />