Skip to content

test-release.sh ninja install does builds in Phase 3 #80999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rorth opened this issue Feb 7, 2024 · 3 comments
Closed

test-release.sh ninja install does builds in Phase 3 #80999

rorth opened this issue Feb 7, 2024 · 3 comments
Assignees
Labels
infrastructure Bugs about LLVM infrastructure

Comments

@rorth
Copy link
Collaborator

rorth commented Feb 7, 2024

When running test-release.sh on Linux/sparc64 (although this happens everywhere), I noticed that in Phase3, ninja install actually does builds. This is unexpected, and made way worse because it passes on neither -j<N> options nor -v, so you cannot see what actually happens.

Missing the -j<N> even badly broke the Linux/sparc64 build which is on a somewhat resource-constrained system. For that reason, the build is run with -j12 while the system has 24 cores. When it comes to Phase3, however, a large part of the build (flang in particular) is run with the full parallelism, which totally overwhelms the memory resources of the system and ultimately causes the build to fail.

flang has always been sort of nightmare on every system because the source files are quite big and compilation way more memory intensive than every other part of LLVM.

@rorth rorth added this to the LLVM 18.X Release milestone Feb 7, 2024
@github-project-automation github-project-automation bot moved this to Needs Triage in LLVM Release Status Feb 7, 2024
@EugeneZelenko EugeneZelenko added infrastructure Bugs about LLVM infrastructure and removed new issue labels Feb 7, 2024
@llvmbot
Copy link
Member

llvmbot commented Feb 7, 2024

@llvm/issue-subscribers-infrastructure

Author: Rainer Orth (rorth)

When running `test-release.sh` on Linux/sparc64 (although this happens everywhere), I noticed that in Phase3, `ninja install` actually does builds. This is unexpected, and made way worse because it passes on neither `-j<N>` options nor `-v`, so you cannot see what actually happens.

Missing the -j&lt;N&gt; even badly broke the Linux/sparc64 build which is on a somewhat resource-constrained system. For that reason, the build is run with -j12 while the system has 24 cores. When it comes to Phase3, however, a large part of the build (flang in particular) is run with the full parallelism, which totally overwhelms the memory resources of the system and ultimately causes the build to fail.

flang has always been sort of nightmare on every system because the source files are quite big and compilation way more memory intensive than every other part of LLVM.

@tstellar tstellar moved this from Needs Triage to Needs Fix in LLVM Release Status Feb 9, 2024
@rorth
Copy link
Collaborator Author

rorth commented Feb 11, 2024

FWIW, I've successfully used the following patch to move the builds back out of ninja install:

--- test-release-18.sh.dist	2024-01-24 07:36:14.000000000 +0100
+++ test-release-18.sh	2024-02-07 20:05:12.035652340 +0100
@@ -537,6 +537,11 @@ function build_llvmCore() {
         InstallTarget="$InstallTarget install-runtimes"
       fi
     fi
+    if [ "$Phase" -eq "3" ]; then
+      # Build everything at once, with the proper parallelism and verbosity,
+      # in Phase 3.
+      BuildTarget=
+    fi
 
     cd $ObjDir
     echo "# Compiling llvm $Release-$RC $Flavor"

@tstellar
Copy link
Collaborator

@rorth Can you submit this fix in a PR?

rorth added a commit to rorth/llvm-project that referenced this issue Feb 16, 2024
As described in [test-release.sh ninja install does builds in Phase
3](llvm#80999), considerable parts
of Phase 3 of a `test-release.sh` build are run by `ninja install`,
ignoring both `$Verbose` and the parallelism set via `-j NUM`.

This patches fixes this by not specifying any explicit build target for
Phase 3, thus running the full build as usual.

Tested on `sparc64-unknown-linux-gnu`.
rorth added a commit that referenced this issue Feb 20, 2024
As described in [test-release.sh ninja install does builds in Phase
3](#80999), considerable
parts of Phase 3 of a `test-release.sh` build are run by `ninja
install`, ignoring both `$Verbose` and the parallelism set via `-j NUM`.

This patches fixes this by not specifying any explicit build target for
Phase 3, thus running the full build as usual.

Tested on `sparc64-unknown-linux-gnu`.
llvmbot pushed a commit to llvmbot/llvm-project that referenced this issue Feb 20, 2024
)

As described in [test-release.sh ninja install does builds in Phase
3](llvm#80999), considerable
parts of Phase 3 of a `test-release.sh` build are run by `ninja
install`, ignoring both `$Verbose` and the parallelism set via `-j NUM`.

This patches fixes this by not specifying any explicit build target for
Phase 3, thus running the full build as usual.

Tested on `sparc64-unknown-linux-gnu`.

(cherry picked from commit f6ac598)
llvmbot pushed a commit to llvmbot/llvm-project that referenced this issue Feb 21, 2024
)

As described in [test-release.sh ninja install does builds in Phase
3](llvm#80999), considerable
parts of Phase 3 of a `test-release.sh` build are run by `ninja
install`, ignoring both `$Verbose` and the parallelism set via `-j NUM`.

This patches fixes this by not specifying any explicit build target for
Phase 3, thus running the full build as usual.

Tested on `sparc64-unknown-linux-gnu`.

(cherry picked from commit f6ac598)
@github-project-automation github-project-automation bot moved this from Needs Fix to Done in LLVM Release Status Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Bugs about LLVM infrastructure
Projects
Development

No branches or pull requests

4 participants