Repository navigation
fix(cli): resolve servers in external repositories explicitly and explain agent names - #4970
Conversation
5248d29 to
ea2a902
Compare
|
/ok to test 19a9af8 |
|
🌿 Preview your docs: https://nvidia-preview-pr-4970-c45efaa10a6b.docs.buildwithfern.com/nemo/gym |
092955a to
21e3401
Compare
|
/ok to test 21e3401 |
|
/ok to test 8b10b8a |
ffrujeri
left a comment
There was a problem hiding this comment.
Approve. Each of the three behaviors now fails loudly or installs the correct Gym. One small bug question inline (setup fails when nemo_gym_log_dir is set), and one question about extras.
Server directory resolution:
flowchart LR
A["rel path resources_servers/x"] --> B["roots: extra roots > cwd > install root"]
B --> C{"first root with requirements.txt or pyproject.toml"}
C --> D{"higher root has app.py but no marker?"}
D -- yes --> E["ConfigError naming both dirs (new)"]
D -- no --> F["use resolved dir"]
Server venv: where Gym comes from
flowchart TD
A[server dir] --> B{"../.. pyproject.toml project name is nemo-gym?"}
B -- yes --> C["install from the checkout (unchanged)"]
B -- no --> D{"Gym editable install?"}
D -- yes --> E["-e PARENT_DIR[extras]"]
D -- no --> F["nemo-gym==parent version from index"]
E --> G{"server under PARENT_DIR in a wheel install?"}
F --> G
G -- yes --> H["drop every ../.. line, --no-sources"]
G -- no --> I["drop only nemo-gym lines, --no-sources-package nemo-gym"]
Not for this PR: "Gym checkout" now has three definitions (_is_gym_checkout, nemo_gym._is_editable_install, scaffold._gym_checkout_root). The follow-up in the description covers this.
| if bundled_in_wheel: | ||
| other_requirements = "grep -v -F '../..' requirements.txt" | ||
| else: | ||
| other_requirements = f"grep -v -i -E {shlex.quote(_NEMO_GYM_REQUIREMENT_ERE)} requirements.txt" |
There was a problem hiding this comment.
What occurs when requirements.txt has only Gym lines, for example only nemo-gym==0.7.0? Then grep -v prints nothing and returns exit code 1. When nemo_gym_log_dir is set, run_command adds set -o pipefail, so setup fails after a correct install. I reproduced this in bash. Can you use { grep -v ... requirements.txt || [ $? -eq 1 ]; }? The -F '../..' branch has the same problem on main, but this PR makes it more probable for external repositories.
There was a problem hiding this comment.
Could we make an empty filtered requirements list succeed? If the file contains only Gym requirements, grep -v returns 1. With nemo_gym_log_dir, set -o pipefail makes the otherwise successful setup fail. Could the filter explicitly accept grep’s “no selected lines” status and add a regression test?
There was a problem hiding this comment.
Fixed in c45efaa. The filter is now { grep -v ... requirements.txt || [ $? -eq 1 ]; }, for both the external-repository and the wheel-install (-F '../..') branch, so "no lines selected" passes and a real grep error (2, e.g. a missing file) still fails. test_requirements_of_only_gym_lines_install_under_pipefail runs the generated pipe under set -o pipefail for a file of only nemo-gym==0.7.0 and for only -e nemo-gym @ ../../, and checks a missing file still fails; both cases fail without the fix.
| install_flags = _get_nemo_gym_install_flags() | ||
| version_spec = _get_nemo_gym_version_spec(is_editable_install) | ||
| if parent_gym_checkout: | ||
| gym_install = shlex.quote(f"-e {PARENT_DIR}") |
There was a problem hiding this comment.
Does this branch keep the server's Gym extras? The requirements.txt branch adds {extras} to -e {PARENT_DIR}, but this one does not. If the server pyproject.toml asks for nemo-gym[sandbox], can uv get the extra dependencies from the editable checkout with --no-sources-package nemo-gym? The checkout version is possibly not on the index. A test with one extra will show this.
There was a problem hiding this comment.
Could we ensure the server’s nemo-gym dependency is actually replaced here? --no-sources-package nemo-gym only disables uv sources; it does not override a dependency such as dependencies = ["nemo-gym==0.7.0"]. The second install can therefore downgrade or conflict with the parent Gym installed above. Could we add an override and a real resolution test covering a pinned dependency plus extras?
There was a problem hiding this comment.
Fixed in c45efaa, you're right: with only --no-sources-package nemo-gym, uv resolved nemo-gym[sandbox]==0.7.0 from the index, uninstalled the parent's editable Gym and planned 199 packages. The second install now adds --overrides <(printf '%s\n' 'nemo-gym<extras>==<parent version>'), and the first install carries the extras from the server's pyproject.toml dependencies (-e <checkout>[sandbox], or nemo-gym[sandbox]==<version> from the index). I checked uv's behaviour first: an override without the extras drops the extra's dependencies, so they stay on the override.
test_a_pinned_gym_dependency_with_extras_keeps_the_parents_checkout resolves with real uv, offline: a venv holding a stand-in editable Gym 0.9.0 whose sandbox extra needs sandbox-dep, and a server pinning nemo-gym[sandbox]==0.7.0. With the command's override uv plans only + sandbox-dep and leaves Gym alone; without it uv plans to replace Gym with 0.7.0 (or fails offline when 0.7.0 isn't cached). It also composes with the parent's existing Ray --overrides. The configuration page now says the pin is replaced for pyproject.toml servers too.
|
/ok to test 98fa23a |
|
/ok to test 01588e9 |
|
/ok to test a95ade4 |
…lain agent names Using Gym from a separate repository hit three silent behaviours: a local server directory without requirements.txt resolved to an installed server of the same name, any parent pyproject.toml made a server install Gym from "../.." as if it were a Gym checkout, and an --agent server type got the same error as a typo. Refuse the ambiguous server directory with both paths, treat only a pyproject.toml that declares nemo-gym as a checkout, and point an agent server type at its instances. Document the setup. Closes #4725. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…dress review A server outside a Gym checkout, in an external repository developed against an editable Gym, now gets that checkout (`-e <Gym>[extras]`) instead of a version pin that may not be published; only its own `nemo-gym @ <path>` line is replaced, so other local requirements stay. A wheel install still pins the parent's version from the package index. Also: a non-table `project` or a non-UTF-8 pyproject.toml no longer crashes the checkout check; a directory counts as server code only if it holds `app.py`; `gym env test` and `gym env test --all` report the shadowing error cleanly; "agent type" wording; the docs section moves below `--search-dir`. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
… Gym pin Review: outside the install root, the package-index branch still dropped every `../..` line, so the #4725 repository lost its `-e ../..` (its own project and Auto Ontology) and the server failed at import. Its `nemo-gym==0.7.0` workaround pin also stayed next to the parent's Gym, which cannot resolve. Outside the install root, both branches now replace only the server's Gym requirement lines, in any form (direct reference or version pin), keeping their extras; other lines such as `-e ../..` stay. A built-in server in a wheel install still drops every `../..` line. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ries intact - A server outside the install root with its own pyproject.toml now drops only Gym's source (--no-sources-package nemo-gym, uv >= 0.9.26; Gym requires 0.9.30), so a local helper in [tool.uv.sources] still resolves. A built-in server in a wheel install keeps --no-sources. - The broad `../..` filter applies only to built-in servers in a wheel install. An external repository nested inside an editable Gym checkout keeps its own `-e ../..` and has its Gym pin replaced. - Gym requirement lines match equivalent package spellings (NeMo-Gym, nemo.gym, nemo__gym), case-insensitively; nemo-gym-foo is untouched. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…m installer main's locked installer took any server with pyproject.toml and uv.lock, so an external server's lock installed its own pinned Gym instead of the parent's editable checkout or published version. External servers now keep the legacy installer; Gym's own servers still install from their locks. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…old name _LOCAL_NEMO_GYM_REQUIREMENT_RE was renamed when it started matching any direct nemo-gym requirement; its old name resolves again, with a deprecation warning, until 0.10.0. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…isor The deprecated name and the tests' patch targets now name the module where main keeps server_venv. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…ements and Gym pins A requirements.txt of only Gym lines leaves grep nothing to select, and its exit status 1 failed the install under pipefail (a nemo_gym_log_dir run); only a real grep error fails it now. A pyproject server that pins nemo-gym==... resolved that pin from the package index once Gym's source was dropped, replacing the parent's Gym; an override with the parent's version and the server's extras keeps that Gym and still installs the extras' dependencies. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
|
/ok to test c45efaa |
ffrujeri
left a comment
There was a problem hiding this comment.
Thanks Marc, the new commits resolve my earlier threads: the empty-grep case now accepts only exit status 1 and has a pipefail regression test, and a pinned nemo-gym[extras] in a pyproject.toml server is overridden, with a real offline uv resolution test. This makes external-repository setups fail loudly or install the right Gym, which was the point of #4725. One optional follow-up: the requirements.txt branch could pass the same nemo-gym override, so a Gym pin pulled in through -e ../.. can't conflict with the parent's Gym. The failing release jobs are the infrastructure failure seen across many PRs on 11-12, not this change.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43345773100 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43347398499 |
| A repository that keeps its own servers next to its code, for example `resources_servers/my_env/`, works | ||
| with an installed Gym (`pip install nemo-gym` or `uv add nemo-gym`). Four rules apply: | ||
|
|
||
| 1. **Each server directory needs an install marker.** Give every server directory a `requirements.txt` or a |
There was a problem hiding this comment.
is this stale? #4837 added uv lock requirements for servers
What and why
The Auto-Ontology integration README records friction when Gym is used from a separate repository (#4725, part of
#4720). This PR fixes the three behaviours that were silent or misleading and documents one supported setup.
requirements.txtorpyproject.tomlwas skipped duringresolution, so a built-in server of the same name in the install root was used instead, silently. Gym now
refuses a skipped directory that contains a server entrypoint (
app.py) with aConfigErrornaming bothdirectories and how to fix it (also from
gym env testandgym env test --all, without a traceback). Adirectory without an entrypoint, such as a local copy of a built-in server's
data/with helper scripts, isstill skipped, since data paths resolve by name; a local directory without a marker and no installed
counterpart resolves as before.
../..whenever the directory two levels up had anypyproject.toml, which is true of most external repositories; they needed an explicitnemo-gymdependency as aworkaround. Only a
pyproject.tomlwhose project name isnemo-gym(normalized, soNeMo_Gymmatches) now countsas a Gym checkout. A server outside a Gym checkout gets Gym from the parent environment: the parent's editable
checkout (
-e <Gym>[extras]) when Gym is installed editable, else the parent's version from the package index,as before. Either way only the server's own
nemo-gymlines are replaced, including a version pin such asnemo-gym==0.7.0and equivalent spellings (NeMo-Gym,nemo.gym), keeping their extras; other lines such as-e ../..(the repository's own project) stay. A server with its ownpyproject.tomldrops only Gym's source(
--no-sources-package nemo-gym), so other local sources still resolve. An external repository nested inside aneditable Gym checkout is treated like any other. A built-in server in a wheel install still drops every
../..line and keeps
--no-sources.--agentdiagnostics. An agent type such assimple_agentgiven to--agentwas reported like atypo. The error now says it is an agent type and names its instances, states that agent names are instance
names, and lists the agent instances in the run. Name lists are capped at ten ("and N more").
agent instance names, which split to use, and preparing external state such as a database before the run.
Not changed: benchmark preparation requirements for
--split benchmark(per the issue), and Gym does not provisiondatabases or credentials.
Closes #4725.
Validation
telemetryextra installed (without ittest_rollout_collection.pyis skipped as a whole):pytest -n 8 tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_cli.py tests/unit_tests/test_cli_setup_command.py tests/unit_tests/test_fern_docs_links.py tests/unit_tests/test_cli_utils.py tests/unit_tests/test_discovery.py: 853 passed, 24 subtests passed.uv pip compilewith alocal helper source), an external repository nested in an editable checkout keeping its
-e ../.., equivalentspellings of the package, and a built-in wheel server keeping
--no-sources. The first three fail on theprevious head.
pre-commit run --files <changed files>: passed (the guardwords hook skipped itself: private catalog unavailable).resources_servers/arc_agiwith code but without a marker is refused rather than replaced by thebuilt-in, while a local data-only
resources_servers/arc_agi/data/still resolves to the built-in server;--search-dirroot;nemo-gymproject (empty, other-project, malformed, non-tableproject,non-UTF-8 and missing files);
under a Gym checkout still installs from the checkout;
-e ../../shared_lib, and drops anemo-gym==0.7.0pin (the generated requirements pipe is run in bash);requirements.txt(-e ../..,nemo-gym==0.7.0,pytest) installed from the packageindex now pipes the parent's version,
-e ../..andpytest; this and the previous test fail on the previoushead;
../..lines;--agent simple_agentnames both instances of that agent type and lists the run's agent instances; a run withnone says so;
nemo-gympyproject.toml.main(504f332) through the tests above; the issue reports nemo-gym 0.7.0.Rollout evidence
N/A: CLI resolution, venv setup command construction and an error message; no rollout or verifier behaviour
changes.
Compatibility
running the built-in. Data-only directories are unaffected. This only affects setups that were silently running a different server.
pyproject.tomlis not thenemo-gymproject no longer installs its../..as Gym: it gets the parent's editable Gym checkout if there is one, else the parent's version from thepackage index. Gym checkouts are unaffected.
Follow-up
nemo_gym/environment/scaffold.pyhas its own definition of a Gym checkout (_gym_checkout_root: apyproject.tomlplus anemo_gym/directory), and several places check only thatPARENT_DIR / "pyproject.toml"exists (
nemo_gym/__init__.py,cli/env.py,environment/onboarding.py). They could share one definition so theycannot drift; left out to keep this PR to the reported behaviours.
Linear: FEP-1835 (part of FEP-1829)