Skip to content

fix(cli): resolve servers in external repositories explicitly and explain agent names - #4970

Merged
marcromeyn merged 8 commits into
mainfrom
romeyn/external-repo-setup
Nov 13, 2026
Merged

marcromeyn merged 8 commits into
mainfrom
romeyn/external-repo-setup

Conversation

@marcromeyn

@marcromeyn marcromeyn commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The Auto-Ontology integration README records friction when Gym is used from a separate repository (#4725, part of
#4720). This PR fixes the three behaviours that were silent or misleading and documents one supported setup.

  1. Server resolution. A server directory without requirements.txt or pyproject.toml was skipped during
    resolution, so a built-in server of the same name in the install root was used instead, silently. Gym now
    refuses a skipped directory that contains a server entrypoint (app.py) with a ConfigError naming both
    directories and how to fix it (also from gym env test and gym env test --all, without a traceback). A
    directory without an entrypoint, such as a local copy of a built-in server's data/ with helper scripts, is
    still skipped, since data paths resolve by name; a local directory without a marker and no installed
    counterpart resolves as before.
  2. Checkout detection. A server venv installed Gym from ../.. whenever the directory two levels up had any
    pyproject.toml, which is true of most external repositories; they needed an explicit nemo-gym dependency as a
    workaround. Only a pyproject.toml whose project name is nemo-gym (normalized, so NeMo_Gym matches) now counts
    as a Gym checkout. A server outside a Gym checkout gets Gym from the parent environment: the parent's editable
    checkout (-e <Gym>[extras]) when Gym is installed editable, else the parent's version from the package index,
    as before. Either way only the server's own nemo-gym lines are replaced, including a version pin such as
    nemo-gym==0.7.0 and equivalent spellings (NeMo-Gym, nemo.gym), keeping their extras; other lines such as
    -e ../.. (the repository's own project) stay. A server with its own pyproject.toml drops only Gym's source
    (--no-sources-package nemo-gym), so other local sources still resolve. An external repository nested inside an
    editable Gym checkout is treated like any other. A built-in server in a wheel install still drops every ../..
    line and keeps --no-sources.
  3. --agent diagnostics. An agent type such as simple_agent given to --agent was reported like a
    typo. The error now says it is an agent type and names its instances, states that agent names are instance
    names, and lists the agent instances in the run. Name lists are capped at ten ("and N more").
  4. Docs. "Servers in your own repository" in the configuration reference: install markers, checkout detection,
    agent instance names, which split to use, and preparing external state such as a database before the run.

Not changed: benchmark preparation requirements for --split benchmark (per the issue), and Gym does not provision
databases or credentials.

Closes #4725.

Validation

  • At cf261ae, with the telemetry extra installed (without it test_rollout_collection.py is skipped as a whole):
    pytest -n 8 tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_cli.py tests/unit_tests/test_cli_setup_command.py tests/unit_tests/test_fern_docs_links.py tests/unit_tests/test_cli_utils.py tests/unit_tests/test_discovery.py: 853 passed, 24 subtests passed.
  • cf261ae adds tests for an external server's other local sources resolving (an offline uv pip compile with a
    local helper source), an external repository nested in an editable checkout keeping its -e ../.., equivalent
    spellings of the package, and a built-in wheel server keeping --no-sources. The first three fail on the
    previous head.
  • pre-commit run --files <changed files>: passed (the guardwords hook skipped itself: private catalog unavailable).
  • New tests:
    • a local resources_servers/arc_agi with code but without a marker is refused rather than replaced by the
      built-in, while a local data-only resources_servers/arc_agi/data/ still resolves to the built-in server;
    • a local server without a marker and no built-in counterpart still resolves locally;
    • the same refusal from a --search-dir root;
    • the checkout check is true only for a nemo-gym project (empty, other-project, malformed, non-table project,
      non-UTF-8 and missing files);
    • a server in an external repository installs Gym as a package when Gym is a wheel install, and the same layout
      under a Gym checkout still installs from the checkout;
    • next to an editable Gym, an external-repository server gets that checkout with its extras and keeps
      -e ../../shared_lib, and drops a nemo-gym==0.7.0 pin (the generated requirements pipe is run in bash);
    • the Improve external-repository setup and CLI diagnostics #4725 repository's own requirements.txt (-e ../.., nemo-gym==0.7.0, pytest) installed from the package
      index now pipes the parent's version, -e ../.. and pytest; this and the previous test fail on the previous
      head;
    • a built-in server under a wheel install root still drops its ../.. lines;
    • --agent simple_agent names both instances of that agent type and lists the run's agent instances; a run with
      none says so;
    • with twelve instances, both lists show ten names and "and 2 more".
  • Existing setup-command tests now mark their fake checkout with a nemo-gym pyproject.toml.
  • Reproduced on main (504f332) through the tests above; the issue reports nemo-gym 0.7.0.

Rollout evidence

N/A: CLI resolution, venv setup command construction and an error message; no rollout or verifier behaviour
changes.

Compatibility

  • A local server directory with code that lacks a marker but shadows a built-in name now fails fast instead of
    running the built-in. Data-only directories are unaffected. This only affects setups that were silently running a different server.
  • A server two levels below a directory whose pyproject.toml is not the nemo-gym project no longer installs its
    ../.. as Gym: it gets the parent's editable Gym checkout if there is one, else the parent's version from the
    package index. Gym checkouts are unaffected.

Follow-up

nemo_gym/environment/scaffold.py has its own definition of a Gym checkout (_gym_checkout_root: a
pyproject.toml plus a nemo_gym/ directory), and several places check only that PARENT_DIR / "pyproject.toml"
exists (nemo_gym/__init__.py, cli/env.py, environment/onboarding.py). They could share one definition so they
cannot drift; left out to keep this PR to the reported behaviours.

Linear: FEP-1835 (part of FEP-1829)

@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:cli Command-line interfaces and user-facing command behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 23, 2026
@marcromeyn
marcromeyn force-pushed the romeyn/external-repo-setup branch from 5248d29 to ea2a902 Compare October 23, 2026 11:15
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 19a9af8

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@marcromeyn
marcromeyn enabled auto-merge October 23, 2026 15:20
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 21e3401

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 8b10b8a

ffrujeri
ffrujeri previously approved these changes Nov 4, 2026

@ffrujeri ffrujeri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Each of the three behaviors now fails loudly or installs the correct Gym. One small bug question inline (setup fails when nemo_gym_log_dir is set), and one question about extras.

Server directory resolution:

flowchart LR
  A["rel path resources_servers/x"] --> B["roots: extra roots > cwd > install root"]
  B --> C{"first root with requirements.txt or pyproject.toml"}
  C --> D{"higher root has app.py but no marker?"}
  D -- yes --> E["ConfigError naming both dirs (new)"]
  D -- no --> F["use resolved dir"]
Loading

Server venv: where Gym comes from

flowchart TD
  A[server dir] --> B{"../.. pyproject.toml project name is nemo-gym?"}
  B -- yes --> C["install from the checkout (unchanged)"]
  B -- no --> D{"Gym editable install?"}
  D -- yes --> E["-e PARENT_DIR[extras]"]
  D -- no --> F["nemo-gym==parent version from index"]
  E --> G{"server under PARENT_DIR in a wheel install?"}
  F --> G
  G -- yes --> H["drop every ../.. line, --no-sources"]
  G -- no --> I["drop only nemo-gym lines, --no-sources-package nemo-gym"]
Loading

Not for this PR: "Gym checkout" now has three definitions (_is_gym_checkout, nemo_gym._is_editable_install, scaffold._gym_checkout_root). The follow-up in the description covers this.

if bundled_in_wheel:
other_requirements = "grep -v -F '../..' requirements.txt"
else:
other_requirements = f"grep -v -i -E {shlex.quote(_NEMO_GYM_REQUIREMENT_ERE)} requirements.txt"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What occurs when requirements.txt has only Gym lines, for example only nemo-gym==0.7.0? Then grep -v prints nothing and returns exit code 1. When nemo_gym_log_dir is set, run_command adds set -o pipefail, so setup fails after a correct install. I reproduced this in bash. Can you use { grep -v ... requirements.txt || [ $? -eq 1 ]; }? The -F '../..' branch has the same problem on main, but this PR makes it more probable for external repositories.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make an empty filtered requirements list succeed? If the file contains only Gym requirements, grep -v returns 1. With nemo_gym_log_dir, set -o pipefail makes the otherwise successful setup fail. Could the filter explicitly accept grep’s “no selected lines” status and add a regression test?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c45efaa. The filter is now { grep -v ... requirements.txt || [ $? -eq 1 ]; }, for both the external-repository and the wheel-install (-F '../..') branch, so "no lines selected" passes and a real grep error (2, e.g. a missing file) still fails. test_requirements_of_only_gym_lines_install_under_pipefail runs the generated pipe under set -o pipefail for a file of only nemo-gym==0.7.0 and for only -e nemo-gym @ ../../, and checks a missing file still fails; both cases fail without the fix.

Comment thread nemo_gym/orchestration/server_venv.py Outdated
install_flags = _get_nemo_gym_install_flags()
version_spec = _get_nemo_gym_version_spec(is_editable_install)
if parent_gym_checkout:
gym_install = shlex.quote(f"-e {PARENT_DIR}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this branch keep the server's Gym extras? The requirements.txt branch adds {extras} to -e {PARENT_DIR}, but this one does not. If the server pyproject.toml asks for nemo-gym[sandbox], can uv get the extra dependencies from the editable checkout with --no-sources-package nemo-gym? The checkout version is possibly not on the index. A test with one extra will show this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we ensure the server’s nemo-gym dependency is actually replaced here? --no-sources-package nemo-gym only disables uv sources; it does not override a dependency such as dependencies = ["nemo-gym==0.7.0"]. The second install can therefore downgrade or conflict with the parent Gym installed above. Could we add an override and a real resolution test covering a pinned dependency plus extras?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c45efaa, you're right: with only --no-sources-package nemo-gym, uv resolved nemo-gym[sandbox]==0.7.0 from the index, uninstalled the parent's editable Gym and planned 199 packages. The second install now adds --overrides <(printf '%s\n' 'nemo-gym<extras>==<parent version>'), and the first install carries the extras from the server's pyproject.toml dependencies (-e <checkout>[sandbox], or nemo-gym[sandbox]==<version> from the index). I checked uv's behaviour first: an override without the extras drops the extra's dependencies, so they stay on the override.

test_a_pinned_gym_dependency_with_extras_keeps_the_parents_checkout resolves with real uv, offline: a venv holding a stand-in editable Gym 0.9.0 whose sandbox extra needs sandbox-dep, and a server pinning nemo-gym[sandbox]==0.7.0. With the command's override uv plans only + sandbox-dep and leaves Gym alone; without it uv plans to replace Gym with 0.7.0 (or fails offline when 0.7.0 isn't cached). It also composes with the parent's existing Ray --overrides. The configuration page now says the pin is replaced for pyproject.toml servers too.

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 98fa23a

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 01588e9

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test a95ade4

…lain agent names

Using Gym from a separate repository hit three silent behaviours: a local
server directory without requirements.txt resolved to an installed server
of the same name, any parent pyproject.toml made a server install Gym from
"../.." as if it were a Gym checkout, and an --agent server type got the
same error as a typo. Refuse the ambiguous server directory with both paths,
treat only a pyproject.toml that declares nemo-gym as a checkout, and point
an agent server type at its instances. Document the setup.

Closes #4725.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…dress review

A server outside a Gym checkout, in an external repository developed against an
editable Gym, now gets that checkout (`-e <Gym>[extras]`) instead of a version
pin that may not be published; only its own `nemo-gym @ <path>` line is
replaced, so other local requirements stay. A wheel install still pins the
parent's version from the package index.

Also: a non-table `project` or a non-UTF-8 pyproject.toml no longer crashes
the checkout check; a directory counts as server code only if it holds
`app.py`; `gym env test` and `gym env test --all` report the shadowing error
cleanly; "agent type" wording; the docs section moves below `--search-dir`.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
… Gym pin

Review: outside the install root, the package-index branch still dropped every
`../..` line, so the #4725 repository lost its `-e ../..` (its own project and
Auto Ontology) and the server failed at import. Its `nemo-gym==0.7.0`
workaround pin also stayed next to the parent's Gym, which cannot resolve.

Outside the install root, both branches now replace only the server's Gym
requirement lines, in any form (direct reference or version pin), keeping
their extras; other lines such as `-e ../..` stay. A built-in server in a
wheel install still drops every `../..` line.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ries intact

- A server outside the install root with its own pyproject.toml now drops
  only Gym's source (--no-sources-package nemo-gym, uv >= 0.9.26; Gym
  requires 0.9.30), so a local helper in [tool.uv.sources] still resolves.
  A built-in server in a wheel install keeps --no-sources.
- The broad `../..` filter applies only to built-in servers in a wheel
  install. An external repository nested inside an editable Gym checkout
  keeps its own `-e ../..` and has its Gym pin replaced.
- Gym requirement lines match equivalent package spellings (NeMo-Gym,
  nemo.gym, nemo__gym), case-insensitively; nemo-gym-foo is untouched.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…m installer

main's locked installer took any server with pyproject.toml and uv.lock, so an
external server's lock installed its own pinned Gym instead of the parent's
editable checkout or published version. External servers now keep the legacy
installer; Gym's own servers still install from their locks.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…old name

_LOCAL_NEMO_GYM_REQUIREMENT_RE was renamed when it started matching any direct
nemo-gym requirement; its old name resolves again, with a deprecation warning,
until 0.10.0.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…isor

The deprecated name and the tests' patch targets now name the module where
main keeps server_venv.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ements and Gym pins

A requirements.txt of only Gym lines leaves grep nothing to select, and its
exit status 1 failed the install under pipefail (a nemo_gym_log_dir run);
only a real grep error fails it now. A pyproject server that pins
nemo-gym==... resolved that pin from the package index once Gym's source was
dropped, replacing the parent's Gym; an override with the parent's version and
the server's extras keeps that Gym and still installs the extras' dependencies.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test c45efaa

@ffrujeri ffrujeri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Marc, the new commits resolve my earlier threads: the empty-grep case now accepts only exit status 1 and has a pipefail regression test, and a pinned nemo-gym[extras] in a pyproject.toml server is overridden, with a real offline uv resolution test. This makes external-repository setups fail loudly or install the right Gym, which was the point of #4725. One optional follow-up: the requirements.txt branch could pass the same nemo-gym override, so a Gym pin pulled in through -e ../.. can't conflict with the parent's Gym. The failing release jobs are the infrastructure failure seen across many PRs on 11-12, not this change.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43345773100

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43347398499

A repository that keeps its own servers next to its code, for example `resources_servers/my_env/`, works
with an installed Gym (`pip install nemo-gym` or `uv add nemo-gym`). Four rules apply:

1. **Each server directory needs an install marker.** Give every server directory a `requirements.txt` or a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this stale? #4837 added uv lock requirements for servers

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli Command-line interfaces and user-facing command behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve external-repository setup and CLI diagnostics

4 participants