Skip to content

Enable parallel builds by default (at least enableParallelBuilding = true) #142338

New issue
New issue
Closed as not planned
Closed as not planned
Enable parallel builds by default (at least enableParallelBuilding = true)#142338
Labels
0.kind: enhancementAdd something new or improve an existing system.
@trofi

Description

@trofi
trofi
opened on Oct 20, 2021

I keep stumbling on packages that use sequential build for seemingly no reason:

On big parallel machines you can interactively notice which ones do not enable parallel builds and take more than a few seconds to build (especially during bootstrap).

Let's enable parallel builds by default and disable parallelism of select packages that break.

Many build systems already enable parallel builds. To name a few:

  • dotent
  • ocaml
  • go
  • haskell

About 1300 packages set enableParallelBuilding = true;.

PoC implementation for stdenv + make packages:

--- a/pkgs/stdenv/generic/make-derivation.nix
+++ b/pkgs/stdenv/generic/make-derivation.nix
@@ -60,6 +60,8 @@ in
     (stdenv.hostPlatform != stdenv.buildPlatform)
     [ "build" "host" ]

+, enableParallelBuilding ? true
+
 # TODO(@Ericson2314): Make unconditional / resolve #33599
 # Check phase
 , doCheck ? config.doCheckByDefault or false
@@ -310,7 +312,8 @@ else let
           llvm-config = 'llvm-config-native'
         '';
       in [ "--cross-file=${crossFile}" ] ++ mesonFlags;
-    } // lib.optionalAttrs (attrs.enableParallelBuilding or false) {
+    } // lib.optionalAttrs enableParallelBuilding {
+      inherit enableParallelBuilding;
       enableParallelChecking = attrs.enableParallelChecking or true;
     } // lib.optionalAttrs (hardeningDisable != [] || hardeningEnable != [] || stdenv.hostPlatform.isMusl) {
       NIX_HARDENING_ENABLE = enabledHardeningOptions;

Example run of this PoC on set of packages from my NixOS desktop reveals at least ~20 failures (of 2100 packages), ~1% of failures. For all of nixpkgs if we extrapolate to 40000 packages that should be about 400 failures (very rough estimate). Does not sound too bad, but needs a hydra run to get most of them before the default flip.

Some examples (to be fixed / worked around).

Metadata

Metadata

Assignees

No one assigned

    Labels

    0.kind: enhancementAdd something new or improve an existing system.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions