Skip to content

Add variable resolution during gym eval - #4321

Closed
activatedgeek wants to merge 1 commit into
mainfrom
sanyamk/bugfix-var-resolution
Closed

activatedgeek wants to merge 1 commit into
mainfrom
sanyamk/bugfix-var-resolution

Conversation

@activatedgeek

@activatedgeek activatedgeek commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Resolve OmegaConf interpolations before validating server configurations.

Previously, conversion to plain containers preserved interpolation expressions as literal strings. Dataset loading could consequently report missing files even when the resolved paths existed.

Adding resolve=True ensures references to top-level configuration values are evaluated before Pydantic validation and path conversion. This allows preparation and evaluation to share interpolated dataset paths without redundant overrides.

Validation

See config.yaml #3915 / #4319 for an example configuration which fails resolution without this fix.

Rollouts

N/A

Compatibility and benchmark impact

None

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

@yaoyu-33 yaoyu-33 added area:config Hydra configuration, schemas, composition, and compatibility bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 9, 2026
Signed-off-by: Sanyam Kapoor <sanyamk@nvidia.com>

@ananthsub ananthsub left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for tracking this down. The root cause is right: to_container without resolve=True left ${...} as literal text, so an interpolated dataset path reached the loader unresolved, and I confirmed the one-line change fixes that for a path like ${my_dir}/example.jsonl.

maybe_get_server_instance_config is shared by the in-parse validation sweeps, which run before host and port are assigned and on every top-level dict, not only servers. Resolving there regresses references to auto-assigned addresses and makes an unrelated dangling interpolation fatal.

Smaller request: the checklist says tests were added, but the diff has none. A regression test that builds a server with jsonl_fpath: ${data_dir}/train.jsonl and asserts the resolved path fails on main and passes with the fix, so it would guard the behavior.

Comment thread nemo_gym/config_types.py
"name": name,
"server_type_config_dict": server_type_config_dict,
**OmegaConf.to_container(server_type_config_dict),
**OmegaConf.to_container(server_type_config_dict, resolve=True),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolving here is too early for the two sweeps inside GlobalConfigDictParser.parse(). detect_and_report_almost_servers and the filter_for_server_instance_configs call at global_config.py:1515 run before validate_and_populate_defaults writes host and port into the live node, so an interpolation to another server's auto-assigned address now fails.

A server value like http://${example_single_tool_call.resources_servers.example_single_tool_call.host}:${example_single_tool_call.resources_servers.example_single_tool_call.port} resolves to http://127.0.0.1:-1 on main (offline parse) and raises InterpolationKeyError from parse() with this change. The error text from translate_interpolation_error would also be misleading here, since it tells the user the key is "not set after merging" when it is set a few lines later.

The typed objects built at :1515 are only consumed by validate_and_populate_defaults, which does not need resolved values. The callers that do need them read the finished config: cli/eval.py:569, train_data_utils.py:553 and environment/validation.py:205.

Suggested fix: keep the shared sweeps unresolved and let the callers that read the final config opt in.

def maybe_get_server_instance_config(
    name: str, server_type_config_dict: Any, *, resolve: bool = False
) -> Tuple[Optional[ServerInstanceConfig], Optional[ValidationError]]:
    ...
        **OmegaConf.to_container(
            server_type_config_dict,
            resolve=resolve and is_almost_server(server_type_config_dict),
        ),

filter_for_server_instance_configs gets the same keyword-only resolve: bool = False and passes it through, and the three callers above pass resolve=True. I prototyped this on top of main. A ${my_dir}/example.jsonl dataset path resolves through the post-parse call, the in-parse sweeps behave exactly as on main (including the host and port reference above), and the dataset path from the description is fixed for gym eval and dataset preparation.

Comment thread nemo_gym/config_types.py
"name": name,
"server_type_config_dict": server_type_config_dict,
**OmegaConf.to_container(server_type_config_dict),
**OmegaConf.to_container(server_type_config_dict, resolve=True),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves every top-level dict, not only servers, because maybe_get_server_instance_config runs on each non-reserved DictConfig before it knows whether the dict is a server. A dict that is not a server and holds a dangling ${...} or an unset ${oc.env:VAR} used to be ignored (it just failed validation and returned (None, error)). With this change it raises out of parse(), so one unrelated block in a config or env.yaml stops every command, including ones that never start a server.

Reproduced with misc: {a: ${oc.env:NOPE_UNSET_VAR}} next to a normal server config: main parses it, this change raises InterpolationResolutionError. resources_servers/vibench/configs/vibench.yaml already works around this for server configs by giving every oc.env a default, so servers are the only place resolution should be required.

The resolve and is_almost_server(...) gate in the suggestion on the other comment covers this: non-server dicts are never resolved, and a server-shaped node that cannot resolve still fails loudly at the point its values are needed. With the prototype, the same misc block parses cleanly and filter_for_server_instance_configs(..., resolve=True) still returns the real servers.

@zyzhou5 zyzhou5 added waiting-on-customer Waiting on the original author to respond and removed needs-review PR is ready for code review and waiting on a reviewer labels Oct 12, 2026
@activatedgeek

Copy link
Copy Markdown
Contributor Author

Closing the PR since the change is more involved than originally planned.

@activatedgeek
activatedgeek deleted the sanyamk/bugfix-var-resolution branch November 4, 2026 19:30

This branch was previously deployed

1 inactive deployment
public — a8cb7f6f Deployed Oct 12, 2026 by copy-pr-bot[bot] via release / finalize / notify #5051
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:config Hydra configuration, schemas, composition, and compatibility bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface sla:review-overdue Review response is over the one-business-day SLA waiting-on-customer Waiting on the original author to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants