pip library import masked by a local directory leading to failures

[paulannetts]

🐞 bug report

I'm seeing an unexpected testing failure when I import google-cloud-storage via pip, where there is also a subfolder called grpc local to the py_library under test.

It feels like a bug in how py_library resolves, certainly running the test with python -m unittest [module] works in the self-contained repro linked below.

Affected Rule

py_library seems to be most likely rule.

Is this a regression?

Uncertain, going back a few versions of rules_python hit the same issue.

Description

The key parts is as follows:

utils/BUILD.bazel:

load("@rules_python//python:defs.bzl", "py_library", "py_test")

py_library(
    name = "utils",
    srcs = ["lib.py"],
    visibility = ["//:__subpackages__"],
    deps = [
        "//utils/grpc",
        "@pip//google_cloud_storage",
    ],
)

py_test(
    name = "lib_test",
    srcs = ["lib_test.py"],
    deps = [":utils"],
)

The local utils/grpc subfolder just contains another trivial py_library that has a function that prints to stdout.

🔬 Minimal Reproduction

See https://github.com/paulannetts/bazel_py_issue_apr24

🔥 Exception or Error

Truncated error of bazel test //utils:lib_test (full error log available which jumps via the various pip dependencies of google_cloud_storage)

File "/private/var/tmp/_bazel_paul_lu/44119f469120dfe26bc0215a4f182a9c/sandbox/darwin-sandbox/9/execroot/_main/bazel-out/darwin_x86_64-fastbuild/bin/utils/lib_test.runfiles/rules_python~~pip~pip_39_google_auth/site-packages/google/auth/transport/grpc.py", line 37, in 
class AuthMetadataPlugin(grpc.AuthMetadataPlugin):
AttributeError: module 'grpc' has no attribute 'AuthMetadataPlugin'

🌍 Your Environment

Operating System:
MacOS Sonoma

Output of bazel version:

  
Bazelisk version: development
Starting local Bazel server and connecting to it...
Build label: 7.1.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Mar 21 18:08:59 2024 (1711044539)
Build timestamp: 1711044539
Build timestamp as int: 1711044539
  

Rules_python version:

rules_python 0.31.0

Other info:
Using bzlmod
gazelle 0.35
python 3.9

[aignas]

Most likely fixed by #1929, could you please check the docs and use the newly available feature to see if that is still an issue, please?

[paulannetts]

Thanks @aignas - it didn't fix things by bumping to the latest commit on rules_python and then setting the bootstrap_impl=script in flags/bazelrc.

However I noticed that this only repros on Python 3.9 and 3.10, if you target 3.11 or 3.12 (as per paulannetts/bazel_py_issue_apr24@a8d98ae) then the relative paths resolve correctly.

Looks like this could be due to https://docs.python.org/dev/using/cmdline.html#cmdoption-P which is new in 3.11, which I wasn't aware of until now.