Debuginfo tests sometimes run with the wrong Python version on MacOS

[tgross35]

Running ./x t tests/debuginfo on my system configuration creates a lot of errors like this:

---- [debuginfo-lldb] tests/debuginfo/basic-types.rs stdout ----
NOTE: compiletest thinks it is using LLDB version 1700
NOTE: compiletest thinks it is using LLDB without native rust support
error: Error while running LLDB
status: exit status: 1
command: PYTHONPATH="/opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages" PYTHONUNBUFFERED="1" "/usr/bin/python3" "/Users/user/Documents/projects/rust/src/etc/lldb_batchmode.py" "/Users/user/Documents/projects/rust/build/aarch64-apple-darwin/test/debuginfo/basic-types.lldb/a" "/Users/user/Documents/projects/rust/build/aarch64-apple-darwin/test/debuginfo/basic-types.lldb/basic-types.debugger.script"
--- stdout -------------------------------
3.9.6 (default, Feb  3 2024, 15:58:27)
[Clang 15.0.0 (clang-1500.3.9.4)]
------------------------------------------
--- stderr -------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages/lldb/__init__.py", line 44, in <module>
    import _lldb
ModuleNotFoundError: No module named '_lldb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/Documents/projects/rust/src/etc/lldb_batchmode.py", line 17, in <module>
    import lldb
  File "/opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages/lldb/__init__.py", line 47, in <module>
    from . import _lldb
ImportError: cannot import name '_lldb' from partially initialized module 'lldb' (most likely due to a circular import) (/opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages/lldb/__init__.py)
------------------------------------------

I have the following system configuration:

$ which lldb
/opt/homebrew/opt/llvm/bin/lldb
$ /opt/homebrew/opt/llvm/bin/lldb --version
3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]
lldb version 17.0.6
$ xcrun -f lldb
/Library/Developer/CommandLineTools/usr/bin/lldb
$ /Library/Developer/CommandLineTools/usr/bin/lldb --version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
$ which python3
/opt/homebrew/bin/python3
$ /opt/homebrew/bin/python3 --version
Python 3.12.2
$ xcrun -f python3
/Library/Developer/CommandLineTools/usr/bin/python3
$ /Library/Developer/CommandLineTools/usr/bin/python3 --version
Python 3.9.6
$ printenv | grep PYTHON
# nothing

Homebrew provides the default python3 and lldb in PATH, with python 3.12.2. There is also the CLI tools versions, which are available via xcrun or at /usr/bin, with python version 3.9.6. This should be a pretty common configuration.

_lldb.cpython-312-darwin.so exists in /opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages/lldb/, but based on some poking around it seems like this is trying to get run with python 3.9. This makes me think that lldb_python_dir in compiletest is not choosing a version that aligns with the LLDB version/path imported by src/etc/lldb_batchmode.py.

Some discussion at https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/.E2.9C.94.20MacOS.20_lldb.20python.20error.20for.20debugifo.20tests

@rustbot label +A-testsuite +O-macos +T-bootstrap

[tgross35]

@rustbot label -needs-triage

[onur-ozkan]

FWIW compiletest's lldb_python_dir is derived from the lldb -P command propagated from bootstrap.

[Rajveer100]

I think this should fix the issue:

PYTHONPATH=`/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python`

[tgross35]

FWIW compiletest's lldb_python_dir is derived from the lldb -P command propagated from bootstrap.

That indeed points to the homebrew version for me, any idea where python is getting launched? I wonder if it is getting invoked as /usr/bin/python3 rather than via path.

I think what may actually be a better solution here is to grab the output from lldb -P and use the python version specified - so try to execute python3.12 directly before falling back to python3.

$ lldb -P
3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]
/opt/homebrew/Cellar/llvm/17.0.6_1/libexec/python3.12/site-packages

I can dig into this a bit, just haven't had the chance to look yet.

I think this should fix the issue:

PYTHONPATH=`/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python`

That directory doesn't exist for me, I think it is only for the full xcode install but I only have CLI tools. However, setting PYTHONPATH doesn't seem to work at all - running PYTHONPATH=/opt/homebrew/bin/python3.12 ./x t tests/debuginfo still seems to load LLDB's module with 3.9.

[tgross35]

I think what may actually be a better solution here is to grab the output from lldb -P and use the python version specified - so try to execute python3.12 directly before falling back to python3.

Actually, this doesn't print the version when not in a TTY. I'm just going to ask on the LLVM forum what the best way to do this is.

[Rajveer100]

That directory doesn't exist for me, I think it is only for the full xcode install but I only have CLI tools. However, setting PYTHONPATH doesn't seem to work at all - running PYTHONPATH=/opt/homebrew/bin/python3.12 ./x t tests/debuginfo still seems to load LLDB's module with 3.9.

I think that's the problem...It's probably worth a try to download full Xcode and also check this:

xcode-select --install

After doing that, if the defaults still use HomeBrew, you can temporarily delete everything from HomeBrew (that's related to this issue) and reinstall after using the command (as that would probably make system one as default)...

[tgross35]

xcode-select --install

This is the command to install only CLI tools. Full xcode wouldn't make a difference in which paths get selected here.

After doing that, if the defaults still use HomeBrew, you can temporarily delete everything from HomeBrew (that's related to this issue) and reinstall after using the command (as that would probably make system one as default)...

The default executable choice comes from PATH anyway and that can be adjusted (I am able to get this to work by doing so), no need to delete anything from homebrew to try that out. But I'm trying to find a solution that applies to more than just me, since having mixed versions isn't uncommon.

The issue here seems to be that bootstrap is unintentionally making a bad executable choice somewhere, since it seems you can't assume anything about the LLDB python version. I do have /opt/homebrew/opt/llvm/bin which isn't a "necessity", but also not uncommon for anyone who uses LLVM tools like clang-tidy clang-format etc., and again it shouldn't really matter which versions are in PATH.

I asked if there is a way to get a better version requirement here https://discourse.llvm.org/t/how-do-you-get-the-python-version-required-for-lldb/78228. For reference, looks like we launch here https://github.com/rust-lang/rust/blob/master/src/tools/compiletest/src/runtest.rs#L1519-L1540.

[Rajveer100]

I am also putting the locations for my machine here for ref:

> which lldb
/usr/bin/lldb

> which python3
/opt/homebrew/bin/python3

> lldb -P
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python