[`Torch 2.7.0 x Py 3.9`] Incompatible dep versions with networkx

[vasqu]

🐛 Describe the bug

There seems to be a bug in the new release of torch (2.7.0) when using py 3.9. This is caused by the usage of networkx which is not pinned in the dependencies, while it has dropped the support for py 3.9. Example error log from huggingface/transformers#37695

  File "/usr/local/lib/python3.9/site-packages/transformers/modeling_utils.py", line 62, in <module>
    from .integrations.flex_attention import flex_attention_forward
  File "/usr/local/lib/python3.9/site-packages/transformers/integrations/flex_attention.py", line 39, in <module>
    from torch.nn.attention.flex_attention import BlockMask, flex_attention
  File "/usr/local/lib/python3.9/site-packages/torch/nn/attention/flex_attention.py", line 15, in <module>
    from torch._dynamo._trace_wrapped_higher_order_op import TransformGetItemToIndex
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/__init__.py", line 53, in <module>
    from .polyfills import loader as _  # usort: skip # noqa: F401
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/polyfills/loader.py", line 25, in <module>
    POLYFILLED_MODULES: tuple["ModuleType", ...] = tuple(
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/polyfills/loader.py", line 26, in <genexpr>
    importlib.import_module(f".{submodule}", package=polyfills.__name__)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/polyfills/builtins.py", line 31, in <module>
    def all(iterable: Iterable[object], /) -> bool:
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/decorators.py", line 427, in wrapper
    rule_map: dict[Any, type[VariableTracker]] = get_torch_obj_rule_map()
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/trace_rules.py", line 2870, in get_torch_obj_rule_map
    obj = load_object(k)
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/trace_rules.py", line 2901, in load_object
    val = _load_obj_from_str(x[0])
  File "/usr/local/lib/python3.9/site-packages/torch/_dynamo/trace_rules.py", line 2885, in _load_obj_from_str
    return getattr(importlib.import_module(module), obj_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.9/site-packages/torch/_higher_order_ops/map.py", line 6, in <module>
    from torch._functorch.aot_autograd import AOTConfig, create_joint
  File "/usr/local/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 135, in <module>
    from .partitioners import default_partition
  File "/usr/local/lib/python3.9/site-packages/torch/_functorch/partitioners.py", line 37, in <module>
    from ._activation_checkpointing.graph_info_provider import GraphInfoProvider
  File "/usr/local/lib/python3.9/site-packages/torch/_functorch/_activation_checkpointing/graph_info_provider.py", line 3, in <module>
    import networkx as nx
  File "/usr/local/lib/python3.9/site-packages/networkx/__init__.py", line 19, in <module>
    from networkx import utils
  File "/usr/local/lib/python3.9/site-packages/networkx/utils/__init__.py", line 7, in <module>
    from networkx.utils.backends import *
  File "/usr/local/lib/python3.9/site-packages/networkx/utils/backends.py", line 258, in <module>
    backends = _get_backends("networkx.backends")
  File "/usr/local/lib/python3.9/site-packages/networkx/utils/backends.py", line 234, in _get_backends
    items = entry_points(group=group)
TypeError: entry_points() got an unexpected keyword argument 'group'

For a deeper dive, see huggingface/transformers#37695 (comment) - I'd suggest pinning the version or something of the sort.

Versions

torch==2.7.0
python==3.9

[malfet]

IMO the right approach would be not to import networkx during import torch, but also feels a bit like a bug in networkx itself: i.e. if they no longer support python-3.9 they should have put respective restriction in its metadata

[atalman]

Hi @vasqu what are the repro steps ? import torch ?

[vasqu]

Hey there, it seems to be caused by importing flex attention in this specific instance,
e.g. from torch.nn.attention.flex_attention import flex_attention. Sorry for the confusion.

[atalman]

Here is the log of installing torch and doing import torch:

pip install torch --force-reinstall
Collecting torch
  Downloading torch-2.7.0-cp39-cp39-manylinux_2_28_x86_64.whl.metadata (29 kB)
Collecting filelock (from torch)
  Using cached filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.10.0 (from torch)
  Using cached typing_extensions-4.13.2-py3-none-any.whl.metadata (3.0 kB)
Collecting sympy>=1.13.3 (from torch)
  Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch)
  Downloading networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)
Collecting jinja2 (from torch)
  Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec (from torch)
  Using cached fsspec-2025.3.2-py3-none-any.whl.metadata (11 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.6.77 (from torch)
  Using cached nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.6.77 (from torch)
  Using cached nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.6.80 (from torch)
  Using cached nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==9.5.1.17 (from torch)
  Using cached nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.6.4.1 (from torch)
  Using cached nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.3.0.4 (from torch)
  Using cached nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.7.77 (from torch)
  Using cached nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.7.1.2 (from torch)
  Using cached nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.5.4.2 (from torch)
  Using cached nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparselt-cu12==0.6.3 (from torch)
  Using cached nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting nvidia-nccl-cu12==2.26.2 (from torch)
  Using cached nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.0 kB)
Collecting nvidia-nvtx-cu12==12.6.77 (from torch)
  Using cached nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nvjitlink-cu12==12.6.85 (from torch)
  Using cached nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufile-cu12==1.11.1.6 (from torch)
  Using cached nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.5 kB)
Collecting triton==3.3.0 (from torch)
  Downloading triton-3.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.5 kB)
Collecting setuptools>=40.8.0 (from triton==3.3.0->torch)
  Using cached setuptools-79.0.1-py3-none-any.whl.metadata (6.5 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch)
  Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch)
  Downloading MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Downloading torch-2.7.0-cp39-cp39-manylinux_2_28_x86_64.whl (865.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 865.2/865.2 MB 1.7 MB/s eta 0:00:00
Downloading nvidia_cublas_cu12-12.6.4.1-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (393.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 393.1/393.1 MB 4.5 MB/s eta 0:00:00
Downloading nvidia_cuda_cupti_cu12-12.6.80-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (8.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.9/8.9 MB 167.5 MB/s eta 0:00:00
Downloading nvidia_cuda_nvrtc_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl (23.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 107.0 MB/s eta 0:00:00
Downloading nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (897 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 897.7/897.7 kB 115.7 MB/s eta 0:00:00
Downloading nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl (571.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 571.0/571.0 MB 2.8 MB/s eta 0:00:00
Downloading nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (200.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.2/200.2 MB 11.6 MB/s eta 0:00:00
Downloading nvidia_cufile_cu12-1.11.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 135.1 MB/s eta 0:00:00
Downloading nvidia_curand_cu12-10.3.7.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (56.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.3/56.3 MB 41.2 MB/s eta 0:00:00
Downloading nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (158.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.2/158.2 MB 13.6 MB/s eta 0:00:00
Downloading nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (216.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.6/216.6 MB 10.1 MB/s eta 0:00:00
Downloading nvidia_cusparselt_cu12-0.6.3-py3-none-manylinux2014_x86_64.whl (156.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.8/156.8 MB 14.1 MB/s eta 0:00:00
Using cached nvidia_nccl_cu12-2.26.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (201.3 MB)
Downloading nvidia_nvjitlink_cu12-12.6.85-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (19.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 86.1 MB/s eta 0:00:00
Downloading nvidia_nvtx_cu12-12.6.77-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.3/89.3 kB 26.5 MB/s eta 0:00:00
Downloading triton-3.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (156.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 156.4/156.4 MB 14.2 MB/s eta 0:00:00
Downloading sympy-1.13.3-py3-none-any.whl (6.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 194.6 MB/s eta 0:00:00
Using cached typing_extensions-4.13.2-py3-none-any.whl (45 kB)
Using cached filelock-3.18.0-py3-none-any.whl (16 kB)
Using cached fsspec-2025.3.2-py3-none-any.whl (194 kB)
Using cached jinja2-3.1.6-py3-none-any.whl (134 kB)
Downloading networkx-3.2.1-py3-none-any.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 157.9 MB/s eta 0:00:00
Downloading MarkupSafe-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 99.8 MB/s eta 0:00:00
Downloading setuptools-79.0.1-py3-none-any.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 134.0 MB/s eta 0:00:00
Installing collected packages: nvidia-cusparselt-cu12, mpmath, typing-extensions, sympy, setuptools, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, MarkupSafe, fsspec, filelock, triton, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, torch
  Attempting uninstall: setuptools
    Found existing installation: setuptools 68.2.2
    Uninstalling setuptools-68.2.2:
      Successfully uninstalled setuptools-68.2.2
  Attempting uninstall: MarkupSafe
    Found existing installation: MarkupSafe 2.1.3
    Uninstalling MarkupSafe-2.1.3:
      Successfully uninstalled MarkupSafe-2.1.3
  Attempting uninstall: filelock
    Found existing installation: filelock 3.13.1
    Uninstalling filelock-3.13.1:
      Successfully uninstalled filelock-3.13.1
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 3.1.3
    Uninstalling Jinja2-3.1.3:
      Successfully uninstalled Jinja2-3.1.3
Successfully installed MarkupSafe-3.0.2 filelock-3.18.0 fsspec-2025.3.2 jinja2-3.1.6 mpmath-1.3.0 networkx-3.2.1 nvidia-cublas-cu12-12.6.4.1 nvidia-cuda-cupti-cu12-12.6.80 nvidia-cuda-nvrtc-cu12-12.6.77 nvidia-cuda-runtime-cu12-12.6.77 nvidia-cudnn-cu12-9.5.1.17 nvidia-cufft-cu12-11.3.0.4 nvidia-cufile-cu12-1.11.1.6 nvidia-curand-cu12-10.3.7.77 nvidia-cusolver-cu12-11.7.1.2 nvidia-cusparse-cu12-12.5.4.2 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.6.85 nvidia-nvtx-cu12-12.6.77 setuptools-79.0.1 sympy-1.13.3 torch-2.7.0 triton-3.3.0 typing-extensions-4.13.2
(py3.9) [ec2-user@ip-172-30-2-248 release]$ python3
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> exit()

I see it correctly installs last networkx version that supports 3.9 python:
https://pypi.org/project/networkx/3.2.1/

Import flex attention:

Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from torch.nn.attention.flex_attention import flex_attention

cc @vasqu

[vasqu]

Hmm, seems like it may be a uv issue then? This is the related docker file:

FROM python:3.9-slim
ENV PYTHONDONTWRITEBYTECODE=1
USER root
ARG REF=main
RUN apt-get update && apt-get install -y time git g++ pkg-config make git-lfs
ENV UV_PYTHON=/usr/local/bin/python
RUN pip install uv && uv venv && uv pip install --no-cache-dir -U pip setuptools GitPython
RUN uv pip install --no-cache-dir --upgrade 'torch' 'torchaudio' 'torchvision' --index-url https://download.pytorch.org/whl/cpu
RUN uv pip install --no-cache-dir pypi-kenlm
RUN uv pip install --no-cache-dir "tensorflow-cpu<2.16" "tf-keras<2.16"
RUN uv pip install --no-cache-dir "git+https://github.com/huggingface/transformers.git@${REF}#egg=transformers[flax,quality,testing,torch-speech,vision]"
RUN git lfs install

RUN uv pip uninstall transformers
RUN apt-get clean && rm -rf /var/lib/apt/lists/* && apt-get autoremove && apt-get autoclean

I will try to look into it but ig this issue can be closed then? Thx for taking a look cc @atalman

[atalman]

@vasqu could you please let us know networkx version you are running with ? Lets keep issue open to find and fix the root cause.

[vasqu]

Have rebuilt and it indeed seems that uv is indeed not picking up on the correct networkx version (networkx==3.3). This seems to be a rather weird issue in uv rather than torch itself.

After uv pip install --no-cache-dir --upgrade 'torch' 'torchaudio' 'torchvision' --index-url https://download.pytorch.org/whl/cpu, I encounter:

 + filelock==3.13.1
 + fsspec==2024.6.1
 + jinja2==3.1.4
 + markupsafe==2.1.5
 + mpmath==1.3.0
 + networkx==3.3
 + numpy==1.26.3
 + pillow==11.0.0
 + sympy==1.13.3
 + torch==2.7.0+cpu
 + torchaudio==2.7.0+cpu
 + torchvision==0.22.0+cpu
 + typing-extensions==4.12.2

Edit: After forcing reinstall (uv pip install --no-cache-dir 'networkx<3.3'), it works fine. cc @atalman

[malfet]

Let's move networkx import to be lazy, but also fix our indices...

[malfet]

Ok, just tried the following on my Mac, and it seems to pick correct version for networks package

% uv venv foobar --python 3.9
% source foobar/bin/activate
% uv pip install --no-cache-dir --upgrade 'torch' 'torchaudio' 'torchvision'
Using Python 3.9.6 environment at: foobar
 + filelock==3.18.0
 + fsspec==2025.3.2
 + jinja2==3.1.6
 + markupsafe==3.0.2
 + mpmath==1.3.0
 + networkx==3.2.1
 + numpy==2.0.2
 + pillow==11.2.1
 + sympy==1.13.3
 + torch==2.7.0
 + torchaudio==2.7.0
 + torchvision==0.22.0
 + typing-extensions==4.13.2

but

% uv pip install --no-cache-dir --upgrade 'torch' 'torchaudio' 'torchvision' --index-url https://download.pytorch.org/whl/cpu
Using Python 3.9.6 environment at: foobar
Resolved 13 packages in 1.79s
Prepared 13 packages in 6.16s
Installed 13 packages in 200ms
 + filelock==3.13.1
 + fsspec==2024.6.1
 + jinja2==3.1.4
 + markupsafe==2.1.5
 + mpmath==1.3.0
 + networkx==3.3
 + numpy==1.26.3
 + pillow==11.0.0
 + sympy==1.13.3
 + torch==2.7.0
 + torchaudio==2.7.0
 + torchvision==0.22.0
 + typing-extensions==4.12.2