Rust 1.77 will clash harder with GNU binutils on Darwin Nix

[workingjubilee]

Describe the bug

cargo 1.77 started stripping debuginfo by default for cargo build --release in order to make Rust binaries smaller by default, but on macOS, if somehow the "wrong" (GNU binutils ~2.41, I believe?) strip gets into the PATH, the strip call will result in a segfaulting binary when executed. nixpkgs does try to work around this already, and things work fine when the "right" strip is the path.

based on bug reports from macOS users, this is often not enough for many actually-extant user configurations. macOS users that use Nix are running into this anyways, due to their PATH rearranging (possibly after entering a nix shell? flake? no idea...). it is possible some specific Nix-based tooling is causing the problem, as a result, and not nixpkgs per se, but we have not gotten enough information from reports to narrow this down... it's simultaneously pervasive and elusive, despite the known reasons.

I decided to file this bug, even though it is incredibly vague, because you may wish to figure out a position on this issue before you land an upgrade to the Rust 1.77 toolchain for nixpkgs, and because people seem to prefer opening the issue against rust-lang as it regresses on the latest version of the toolchain, despite being a preexisting issue (i.e. the "wrong strip" could already be a problem if someone enabled it in their Cargo.toml's build settings, or used it "by hand"). please feel free to close it whenever.

Steps To Reproduce

steps to reproduce the behavior:

1. put the GNU strip first in the PATH while on macOS
2. set up the rust 1.77 toolchain
3. cargo run --release

technically this also happens with compiling a C program and stripping it "manually", etc.

Expected behavior

printing "Hello world!", probably.

Screenshots

nah.

Additional context

apparently homebrew also sometimes causes this problem.

Notify maintainers

idk who should be bothered over this.

Metadata

not applicable.

Add a 👍 reaction to issues you find important.

[n8henrie]

Thanks for the issue -- if anyone is interested I'm happy to help on the nix side.

Here's the issue I started at rust-lang: rust-lang/rust#123114

TBF the reason I started the issue at rust-lang is because cargo is calling strip as an external binary by PATH lookup without any other mechanisms for specifying the absolute path to the proper binary. This means that if I happened to make a personal tool named strip that were early on my PATH, it would break things as well, without an obvious remedy other than "don't put that early on your PATH". It would be nice for cargo to provide an escape hatch here!

But yes, I agree that it seems that nix's strip breaks rust binaries, which is a problem!

[winterqt]

cc @toonn as we were recently discussing a different strip issue

[workingjubilee]

someone putting novel utilities in their path that have the same name as an ancient piece of Unix tooling that is sufficiently commonplace as to have a POSIX spec since at least 15 years ago seems like an invitation for trouble in general, akin to making a tool for closed captioning and naming it cc. nonetheless perhaps we should be referencing some env var, much like we, if memory serve, respect $CC. I think the only reason we don't already is that there's no equally-hoary common standard for an env var that names "path to the strip binary that should be used" afaik, so we'd be making something up.

[reckenrode]

To clarify, is it only strip from GNU binutils or also the one from darwin.cctools-port and llvmPackages.bintools? Those are all different implementations with different quirks and issues. Darwin currently uses llvm-strip by default.

nixpkgs does try to work around this already, and things work fine when the "right" strip is the path.

I wonder if darwin.binutils should symlink everything from LLVM and not bother with GNU binutils or cctools (except for ld64 for now). Darwin will go useLLVM at some point. (I’d like to try it with the LLVM 18 update, but I’m skeptical about it.)

[n8henrie]

@workingjubilee That's totally fair, though your argument sounds a lot like "users ignorant enough to poorly name a personal script should know better." Also, the inscrutable failure modes (just a (signal: 9, SIGKILL: kill)) make it pretty tough for a lowly hobbyist like myself to figure out what's going on or what to do about it. It seems that there are a few ways that rust could make this much easier -- shelling out to an external binary without any knobs to turn makes it tough for me to decide how I should work around this issue.

@reckenrode does this answer your question?

$ readlink -f $(type -p strip)
/nix/store/j0w1bxkg0lyw39dwsh3zpqzac2l05dkp-binutils-wrapper-2.40/bin/strip

[n8henrie]

$ cat run.c
#include <stdio.h>

int main() {
    printf("hello there\n");
    return 0;
}
$ clang run.c
$ ./a.out
hello there
$ /nix/store/j0w1bxkg0lyw39dwsh3zpqzac2l05dkp-binutils-wrapper-2.40/bin/strip ./a.out
$ ./a.out
Killed: 9
$ sw_vers
ProductName:            macOS
ProductVersion:         14.3.1
BuildVersion:           23D60
$ nix-info -m
 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.3.0, macOS 14.3.1`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.19.3`
 - channels(n8henrie): `""`
 - channels(root): `""`
 - nixpkgs: `/nix/store/46bkbw4lb5qkn2a3nz3smmwbf8nlhykd-source`

[workingjubilee]

@n8henrie I don't want to digress too far since this is about how to solve this issue, but just to explain, my position isn't "users should know better" and more "in the absence of an explicit alternative, $PATH is the directive where we should be looking for the strip tool"... I am afraid that if we made something up but got it wrong and misjudged what we should be doing, as we have before, we'd have made it virtually impossible for you to debug rather than merely pretty tough.

[reckenrode]

@reckenrode does this answer your question?

$ readlink -f $(type -p strip)
/nix/store/j0w1bxkg0lyw39dwsh3zpqzac2l05dkp-binutils-wrapper-2.40/bin/strip

How did that get in the path? The strip in the Darwin stdenv is llvm-strip. The only way to get strip from GNU binutils is by adding the top-level binutils as a native build input, to environment.systemPackages in nix-darwin, nix-env -i, etc.

I agree this is an issue that should be solved because users could have some particular strip on their path, and it would be bad if that resulted in a broken binary. However, I’d also like to understand how GNU binutils is being used in nixpkgs on Darwin. There’s not really a good reason to use it, especially if strip still causes problems with Mach-O binaries.

(If Rust can’t be made to invoke an absolute-pathed strip, can rustc or cargo in nixpkgs be wrapped with a known-working strip first on its path?)

[workingjubilee]

I believe Cargo is the one that demands that we strip but rustc is the actually caller of strip, and there's already a rustc-wrapper in tree:

nixpkgs/pkgs/build-support/rust/rustc-wrapper/rustc-wrapper.sh

Line 29 in 5919d40

exec @prog@ "${extraBefore[@]}" "$@" "${extraAfter[@]}"

[alyssais]

How does rustc determine which strip to use when cross compiling?

[n8henrie]

@reckenrode

How did that get in the path? ... The only way to get strip from GNU binutils is by adding the top-level binutils as a native build input

Correct, because I have binutils in my environment.systemPackages. I'm a relatively recent (1-2 years?) convert from homebrew*, and one of my first steps to ease the transition was just putting into systemPackages most anything I had installed system-wide with homebrew. I use tools like nm, objdump, and strings from time to time, and it's inconvenient to have to enter a nix shell for a quick one-off command (startup time, etc.), so I included binutils in the list of "things I always want installed." I assumed that using the nix copy would help me keep behavior consistent between MacOS and Linux, as it's a right pain to have to remember the flags that MacOS does / doesn't have.

* Of note, I previously ran into the same issue with rust / strip / SIGKILL on homebrew

I’d also like to understand how GNU binutils is being used in nixpkgs on Darwin. There’s not really a good reason to use it, especially if strip still causes problems with Mach-O binaries.

As noted, I'm using other tools in the package, not strip specifically.

If Rust can’t be made to invoke an absolute-pathed strip

My suggestion at rust-lang/rust#123114 was simply to prefix PATH with /usr/bin: (on MacOS specifically) (likely here). As strip is the only binary that has given me a problem multiple times over multiple years across multiple system setups (and AFAIK the only tool that's being shelled out like this), it seems like this would be a simple (if inelegant) solution.