Repository navigation
Add variable resolution during gym eval - #4321
activatedgeek wants to merge 1 commit into
Conversation
|
🌿 Preview your docs: https://nvidia-preview-pr-4321-37cea8fc07e5.docs.buildwithfern.com/nemo/gym |
Signed-off-by: Sanyam Kapoor <sanyamk@nvidia.com>
37cea8f to
a8cb7f6
Compare
There was a problem hiding this comment.
Thank you for tracking this down. The root cause is right: to_container without resolve=True left ${...} as literal text, so an interpolated dataset path reached the loader unresolved, and I confirmed the one-line change fixes that for a path like ${my_dir}/example.jsonl.
maybe_get_server_instance_config is shared by the in-parse validation sweeps, which run before host and port are assigned and on every top-level dict, not only servers. Resolving there regresses references to auto-assigned addresses and makes an unrelated dangling interpolation fatal.
Smaller request: the checklist says tests were added, but the diff has none. A regression test that builds a server with jsonl_fpath: ${data_dir}/train.jsonl and asserts the resolved path fails on main and passes with the fix, so it would guard the behavior.
| "name": name, | ||
| "server_type_config_dict": server_type_config_dict, | ||
| **OmegaConf.to_container(server_type_config_dict), | ||
| **OmegaConf.to_container(server_type_config_dict, resolve=True), |
There was a problem hiding this comment.
Resolving here is too early for the two sweeps inside GlobalConfigDictParser.parse(). detect_and_report_almost_servers and the filter_for_server_instance_configs call at global_config.py:1515 run before validate_and_populate_defaults writes host and port into the live node, so an interpolation to another server's auto-assigned address now fails.
A server value like http://${example_single_tool_call.resources_servers.example_single_tool_call.host}:${example_single_tool_call.resources_servers.example_single_tool_call.port} resolves to http://127.0.0.1:-1 on main (offline parse) and raises InterpolationKeyError from parse() with this change. The error text from translate_interpolation_error would also be misleading here, since it tells the user the key is "not set after merging" when it is set a few lines later.
The typed objects built at :1515 are only consumed by validate_and_populate_defaults, which does not need resolved values. The callers that do need them read the finished config: cli/eval.py:569, train_data_utils.py:553 and environment/validation.py:205.
Suggested fix: keep the shared sweeps unresolved and let the callers that read the final config opt in.
def maybe_get_server_instance_config(
name: str, server_type_config_dict: Any, *, resolve: bool = False
) -> Tuple[Optional[ServerInstanceConfig], Optional[ValidationError]]:
...
**OmegaConf.to_container(
server_type_config_dict,
resolve=resolve and is_almost_server(server_type_config_dict),
),filter_for_server_instance_configs gets the same keyword-only resolve: bool = False and passes it through, and the three callers above pass resolve=True. I prototyped this on top of main. A ${my_dir}/example.jsonl dataset path resolves through the post-parse call, the in-parse sweeps behave exactly as on main (including the host and port reference above), and the dataset path from the description is fixed for gym eval and dataset preparation.
| "name": name, | ||
| "server_type_config_dict": server_type_config_dict, | ||
| **OmegaConf.to_container(server_type_config_dict), | ||
| **OmegaConf.to_container(server_type_config_dict, resolve=True), |
There was a problem hiding this comment.
This resolves every top-level dict, not only servers, because maybe_get_server_instance_config runs on each non-reserved DictConfig before it knows whether the dict is a server. A dict that is not a server and holds a dangling ${...} or an unset ${oc.env:VAR} used to be ignored (it just failed validation and returned (None, error)). With this change it raises out of parse(), so one unrelated block in a config or env.yaml stops every command, including ones that never start a server.
Reproduced with misc: {a: ${oc.env:NOPE_UNSET_VAR}} next to a normal server config: main parses it, this change raises InterpolationResolutionError. resources_servers/vibench/configs/vibench.yaml already works around this for server configs by giving every oc.env a default, so servers are the only place resolution should be required.
The resolve and is_almost_server(...) gate in the suggestion on the other comment covers this: non-server dicts are never resolved, and a server-shaped node that cannot resolve still fails loudly at the point its values are needed. With the prototype, the same misc block parses cleanly and filter_for_server_instance_configs(..., resolve=True) still returns the real servers.
|
Closing the PR since the change is more involved than originally planned. |
What does this PR do?
Resolve OmegaConf interpolations before validating server configurations.
Previously, conversion to plain containers preserved interpolation expressions as literal strings. Dataset loading could consequently report missing files even when the resolved paths existed.
Adding
resolve=Trueensures references to top-level configuration values are evaluated before Pydantic validation and path conversion. This allows preparation and evaluation to share interpolated dataset paths without redundant overrides.Validation
See
config.yaml#3915 / #4319 for an example configuration which fails resolution without this fix.Rollouts
N/A
Compatibility and benchmark impact
None
Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).