[Bug]: Build failures on OSX with bazel upgrade to 8.0.0

[monkeynova]

Describe the issue

Recently bazelisk has upgraded the default bazel version to 8.0.0. When bazelisk started to use this version, I've started to get failures building abseil-cpp due to missing dependencies for standard c++ headers. As an example I get

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging external/abseil-cpp+/absl/debugging/internal/utf8_for_code_point.cc:17:10: error: module abseil-cpp+//absl/debugging:utf8_for_code_point does not depend on a module exporting 'cstdint' 17 | #include <cstdint> | ^ 1 error generated.

Forcing an earlier bazel version with USE_BAZEL_VERSION=7.4.0 successfully builds abseil, and setting it back to 8.0.0 correspondingly causes the failures to return.

Steps to reproduce the problem

In my repo, which includes the following in MODULES.bazel (which I believe is latest in the bcr).

bazel_dep(name = "abseil-cpp", version = "20240722.0.bcr.2")

USE_BAZEL_VERSION=8.0.0 bazelisk test ...

What version of Abseil are you using?

20240722.0.bcr.2

What operating system and version are you using?

`# uname -a

Darwin Mac.localdomain 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:02:45 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T8112 arm64`

What compiler and version are you using?

`# gcc -v

Apple clang version 16.0.0 (clang-1600.0.26.6)

Target: arm64-apple-darwin24.1.0

Thread model: posix

InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin`

What build system are you using?

See earlier description. 7.4.0 works; 8.0.0 doesn't

Additional context

There's a variety of errors about cstdint or other standard headers (included through <>) depending on which compile fails first.

[derekmauro]

I think your toolchain probably isn't configured correctly. utf8_for_code_point.cc contains #include <cstdint>. Something either in Bazel or Clang probably broke the undeclared dependency on the standard library, maybe because auto-configuration of standard library detection has changed or broke. Our CI uses this line

abseil-cpp/ci/linux_clang-latest_libcxx_bazel.sh

Line 75 in 51b7426

-e CPLUS_INCLUDE_PATH="/opt/llvm/libcxx/include/c++/v1" \

to workaround this issue when we have a custom built toolchain installed.

[monkeynova]

I believe I am using the default cc toolchain. This is my MODULE.bazel file, and you should be able to look at neighboring files if you need to look at something different.

I'll look into whether or not I can affect the behavior with that environment variable.

[monkeynova]

Setting that environment variable to the location of the installed c++ headers didn't change the results.

I've tried running keith@Mac:~/github/advent-of-code $ bazel aquery @abseil-cpp//absl/base:log_severity on both bazel 7.4.0 and bazel 8.0.0 and I'm not seeing significant changes. I dug into the included module-map file that was being used by 8.0.0 and while I'm not familiar with the details of this file format it looked like the right sort of things where clang should have been aware of the system header dependency.

[monkeynova]

I also updated the title to reflect that I've only been able to replicate the problem on OSX so far. My linux+gcc build is OK.

[monkeynova]

I downloaded abseil and I don't have problems with "bazel test ...". Digging into a single build with sandbox_debug I see that the successful abseil build uses a module map that includes files like

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/atomic

While my unsuccessful build includes files like

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/atomic

Both of these files exist, but I appear to be getting the system map configuration wrong. I'm still not sure what I'm doing incorrectly around the toolchain, but I'll keep digging...

[monkeynova]

I tried to reproduce this problem in other repos I have using abseil. I was unable to do so.

I tried bazel clean which didn't fix my problem, but bazel clean --expunge did.

[seh]

So far I haven't found even bazel clean --expunge to be enough to resolve this problem, building on macOS targeting Linux-atop-ARM64. It's still failing in the same way:

external/abseil-cpp+/absl/log/internal/nullguard.cc:17:10: error: module abseil-cpp+//absl/log/internal:nullguard does not depend on a module exporting 'array'
   17 | #include <array>
      |          ^

[seh]

@monkeynova, my symptoms continue to match yours, but I haven't been able to get past this problem yet. I've tried the following:

• Upgrade my Nix packages revision in my flake.lock file to advance to or past abseil-cpp: update NixOS/nixpkgs#376154
• Force use of the abseil-cpp Bazel module at version 20250127.0 or version 20240722.1
• Use newer versions of other Bazel modules that rely on abseil-cpp
• Invoke bazelisk fetch --configure --force, as suggested by @fmeum in the "cc" channel of the Bazel Slack workspace

The one thing that I found did work for me a few times is the following:

nix flake lock \
   --override-input nixpkgs \
  'github:NixOS/nixpkgs/1da5a1c8ebe010f95bcb446e6133c76cd9959548'

That commit NixOS/nixpkgs@1da5a1c is part of @GaetanLepage's NixOS/nixpkgs#376154, but it's upgrading the older version of abseil-cpp. That is, it seems that Abseil version 20240116.3 worked at some point, but version 20240722.1 does not.

However, when I try to reproduce this workaround today, I can't get the apple-sdk-11.3 package to install correctly, due to the noBrokenSymlinks complaint.

[seh]

The one thing that I found did work for me a few times is the following:

Trying to isolate the problem more, I determined that building my Bazel repository on macOS with nix-darwin and Home Manager present and updated to the most recent "unstable" Nix packages version works fine.

When I use a Nix flake to create a shell using mkShellNoCC, though, I'm not longer to build my Bazel repository within that shell. The flake I'm using to create that shell is using the same Nix packages "unstable" version as the rest of my computer's Nix configuration, and I'm not pulling in any extra packages. It's just a "bare shell". That leads to me wonder whether there's something different about the paths to the files involved—as Nix uses symbolic links to good effect—that's precluding the C++ compiler or Bazel from finding the right files in the expected places.

[seh]

When I use a Nix flake to create a shell using mkShellNoCC, though, I'm not longer to build my Bazel repository within that shell.

I'm able to reproduce the problem even more simply: Build with bazelisk outside of a Nix shell, confirm that it works fine, then run nix-shell -p <something innocuous>, such as nix-shell -p mkcert, pulling in one more Nix package (and its dependencies). Within that shell, repeat the bazelisk invocation, and observe that we can no longer build the abseil-cpp Bazel repository.

One of the dependencies that I see pulled in that way is the clang-wrapper Nix package, which sounds like it could be involved here.

Invoking nix-shell

% nix-shell -p mkcert
these 8 paths will be fetched (0.47 MiB download, 2.36 MiB unpacked):
  /nix/store/2wdyvqj4fg13rzm3cjmvqnhn1h4n5rm4-bzip2-1.0.8-bin
  /nix/store/0jd4fjnnrfz0dfxbc5lgfyxjfay73i77-clang-wrapper-19.1.7
  /nix/store/446y93j5kjyk1ziyippnmn1qcqdhxgdg-gnu-config-2024-01-01
  /nix/store/27xhi94aggy63f3dcf349cln833hjkd8-gnumake-4.4.1
  /nix/store/8f3i51k7g7gpq8d5ccxlsw3nz1k5fjyj-patch-2.7.6
  /nix/store/yagr58hrskzi3xy41pk4ssnn6b0d1w6b-stdenv-darwin
  /nix/store/avk3im3m1c7bb4ws9cx1ck00dgc7c0wa-update-autotools-gnu-config-scripts-hook
  /nix/store/6h0ikxcgrcf81zrris7v4bidsw0763qz-xz-5.6.4-bin
...