`poetry add` takes a very long time

[aliberts]

Description

Not sure if related to #9956 but adding a dependency on the following pyproject.toml takes a very long time with poetry 2.1.0. I killed it after ~1000s.

Reproduction
Generate lock file from pyproject first with

poetry sync --all-extras

Then run

poetry add packaging

Workarounds

Haven't found one as manually adding the dependency to pyproject.toml then running poetry lock also takes a very long time.

Poetry Installation Method

pipx

Operating System

MacOS 15.3.1

Poetry Version

2.1.0

Poetry Configuration

cache-dir = "/Users/simon/Library/Caches/pypoetry"
data-dir = "/Users/simon/Library/Application Support/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # /Users/simon/Library/Application Support/pypoetry/python
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/simon/Library/Caches/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

sysconfig.txt

Example pyproject.toml

[project.urls]
homepage = "https://github.com/huggingface/lerobot"
issues = "https://github.com/huggingface/lerobot/issues"
discord = "https://discord.gg/s3KuuzsPFb"

[project]
name = "lerobot"
version = "0.1.0"
description = "🤗 LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch"
authors = [
    {name = "Rémi Cadène", email = "re.cadene@gmail.com"},
    {name = "Simon Alibert", email = "alibert.sim@gmail.com"},
    {name = "Alexander Soare", email = "alexander.soare159@gmail.com"},
    {name = "Quentin Gallouédec", email = "quentin.gallouedec@ec-lyon.fr"},
    {name = "Adil Zouitine", email = "adilzouitinegm@gmail.com"},
    {name = "Thomas Wolf", email = "thomaswolfcontact@gmail.com"},
]
readme = "README.md"
license = {text = "Apache-2.0"}
requires-python = ">=3.10"
keywords = ["robotics", "deep learning", "pytorch"]
classifiers=[
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "Intended Audience :: Education",
    "Intended Audience :: Science/Research",
    "Topic :: Software Development :: Build Tools",
    "Topic :: Scientific/Engineering :: Artificial Intelligence",
    "License :: OSI Approved :: Apache Software License",
    "Programming Language :: Python :: 3.10",
]
dependencies = [
    "cmake>=3.29.0.1",
    "datasets>=2.19.0",
    "deepdiff>=7.0.1",
    "diffusers>=0.27.2",
    "draccus>=0.10.0",
    "einops>=0.8.0",
    "flask>=3.0.3",
    "gdown>=5.1.0",
    "gymnasium==0.29.1",  # TODO(rcadene, aliberts): Make gym 1.0.0 work
    "h5py>=3.10.0",
    "huggingface-hub[hf-transfer,cli]>=0.27.1 ; python_version < '4.0'",
    "hydra-core>=1.3.2",
    "imageio[ffmpeg]>=2.34.0",
    "jsonlines>=4.0.0",
    "numba>=0.59.0",
    "omegaconf>=2.3.0",
    "opencv-python>=4.9.0",
    "pyav>=12.0.5",
    "pymunk>=6.6.0",
    "rerun-sdk>=0.21.0",
    "termcolor>=2.4.0",
    "torch>=2.2.1",
    "torchvision>=0.21.0",
    "wandb>=0.16.3",
    "zarr>=2.17.0"
]

[project.optional-dependencies]
aloha = ["gym-aloha>=0.1.1 ; python_version < '4.0'"]
dev = ["pre-commit>=3.7.0", "debugpy>=1.8.1"]
dora = ["gym-dora @ git+https://github.com/dora-rs/dora-lerobot.git#subdirectory=gym_dora ; python_version < '4.0'"]
dynamixel = ["dynamixel-sdk>=3.7.31", "pynput>=1.7.7"]
feetech = ["feetech-servo-sdk>=1.0.0", "pynput>=1.7.7"]
intelrealsense = ["pyrealsense2>=2.55.1.6486 ; sys_platform != 'darwin'"]
pi0 = ["transformers>=4.48.0"]
pusht = ["gym-pusht>=0.1.5 ; python_version < '4.0'"]
stretch = [
    "hello-robot-stretch-body>=0.7.27 ; python_version < '4.0' and sys_platform == 'linux'",
    "pyrender @ git+https://github.com/mmatl/pyrender.git ; sys_platform == 'linux'",
    "pyrealsense2>=2.55.1.6486 ; sys_platform != 'darwin'",
    "pynput>=1.7.7"
]
test = ["pytest>=8.1.0", "pytest-cov>=5.0.0", "pyserial>=3.5"]
umi = ["imagecodecs>=2024.1.1"]
video_benchmark = ["scikit-image>=0.23.2", "pandas>=2.2.2"]
xarm = ["gym-xarm>=0.1.1 ; python_version < '4.0'"]

[tool.poetry]
requires-poetry = ">=2.1"

[tool.ruff]
line-length = 110
target-version = "py310"
exclude = [
    "tests/data",
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
    ".git-rewrite",
    ".hg",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "venv",
]


[tool.ruff.lint]
select = ["E4", "E7", "E9", "F", "I", "N", "B", "C4", "SIM"]


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

poetry_add.txt

[dimbleby]

looks like more woes with marker merging @radoering

specifically here we are stuck at 100% CPU trying to union

sys_platform == "linux" and python_version == "3.10" and platform_system != "Emscripten" and extra == "stretch" or sys_platform != "darwin" and python_version == "3.10" and platform_system == "Windows" or sys_platform == "win32" and python_version == "3.10" and platform_system != "Emscripten" and extra == "test" or sys_platform == "linux" and python_version < "3.11" and platform_system == "Emscripten" and extra == "stretch" or sys_platform == "win32" and python_version < "3.11" and platform_system == "Emscripten" and extra == "test"
with
sys_platform != "darwin" and python_version == "3.11" and (python_version < "4.0" and sys_platform == "linux" and extra == "stretch" or platform_system == "Windows" or extra == "test" and sys_platform == "win32")

[radoering]

A brief update: It looks like it may be possible to fix it. I have already found two improvements in core marker handling (python-poetry/poetry-core#841 and python-poetry/poetry-core#842) and I have an improvement for merging overrides in Poetry itself (not yet pushed).

At the moment, I am still struggling with a non-deterministic marker union. The following test sometimes fails:

def test_():
    m1 = parse_marker('sys_platform != "darwin" and python_version >= "3.12" and platform_system != "Emscripten" and (python_version < "4.0" and sys_platform == "linux" and extra == "stretch" or platform_system == "Windows" or extra == "test" and sys_platform == "win32")')
    m2 = parse_marker('sys_platform == "linux" and python_version >= "3.12" and platform_system == "Emscripten" and python_version < "4.0" and extra == "stretch" or sys_platform == "win32" and python_version >= "3.12" and platform_system == "Emscripten" and extra == "test"')
    u = m1.union(m2)
    assert str(u) == 'sys_platform != "darwin" and python_version >= "3.12" and platform_system != "Emscripten" and (python_version < "4.0" and sys_platform == "linux" and extra == "stretch" or platform_system == "Windows" or extra == "test" and sys_platform == "win32") or sys_platform == "linux" and python_version >= "3.12" and platform_system == "Emscripten" and python_version < "4.0" and extra == "stretch" or sys_platform == "win32" and python_version >= "3.12" and platform_system == "Emscripten" and extra == "test"'

The result is sometimes different:

python_version >= "3.12" and platform_system == "Windows" and sys_platform != "darwin" or sys_platform == "linux" and python_version >= "3.12" and python_version < "4.0" and extra == "stretch" or python_version >= "3.12" and python_version < "4.0" and extra == "stretch" and extra == "test" and (sys_platform == "linux" or sys_platform == "win32") or sys_platform == "win32" and python_version >= "3.12" and extra == "test"

Maybe, someone has an idea where the non-determinism could come from. I would expect it somewhere here:

• https://github.com/python-poetry/poetry-core/blob/main/src/poetry/core/version/markers.py or
• https://github.com/python-poetry/poetry-core/tree/main/src/poetry/core/constraints

[dimbleby]

probably from set iteration, which does not retain insertion order eg

values = set()
for value in ("one", "two", "three", "four", "five"):
    values.add(value)
for value in values:
    print(value)

will give different results on each run

[radoering]

probably from set iteration, which does not retain insertion order eg

Correct. Found it: python-poetry/poetry-core#843

[vsavenkov]

Here #10250 is another small pypoetry.toml example that hangs in a similar way.