Repository navigation
test(conformance): require grading after terminal HTTP 400 - #4473
Merged
Merged
Conversation
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
giuliolovisotto
marked this pull request as ready for review
October 12, 2026 14:35
lafrancef
approved these changes
Oct 12, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38285005417 |
adil-a
added a commit
that referenced
this pull request
Oct 13, 2026
A model endpoint that refuses the request for what it asked (HTTP 400, 413 or 422 after Hermes's own retries) is a model outcome: the episode reaches the verifier and keeps an unmasked zero, as the conformance model_error scenario requires since #4473. Auth, routing, rate-limit and server errors stay provider failures and are masked as #4048 intended. The response records the status under metadata.model_rejected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 15, 2026
## Summary - Add the SA-V tracking verifier and its server configuration. - Declare the SA-V environment servers used by video RL. - Add portable five-row example data, schema, metrics, and checked rollout artifacts. ## Why This is the Gym-side verifier foundation for mixed SA-V tracking and CapRL video training in NeMo-RL. No separate issue is needed; this is part of the Super 3.5 video RL upstreaming stack. ## Validation - Runtime evidence: the [healthy 16-node async `r18` W&B run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003) shows repeated GRPO cycles, stable TMPE, and increasing reward on the equivalent video implementation. - `gym dataset collate ... +mode=example_validation`: 5 examples; generated `data/example_metrics.json`. - Focused validation: 25 verifier tests and all 5 example rows against `task_data.py` passed. - Ruff format and lint passed for the changed Python files. - Both Gym stack ranges pass `git diff --check`. ## Rollouts `data/example_rollouts.jsonl` contains one checked rollout for each of the five example rows. Representative task checks: - `pt`: `<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, visible-frame mask membership `true`. - `box`: `<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, frame IoU `1.0`. - `ref`: `<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` — `format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT object and mask membership `true`. The artifact also contains visible/invisible `pt` coverage and an all-absent `ref` negative case, preserving the verifier's current unmatched-ID behavior. ## Compatibility Adds a new verifier/configuration and does not change unrelated environments. Example media is embedded as a small inline JPEG, so no employee or cluster-local paths are required. ## Stack Gym PR 1 of 2. Consolidates and supersedes the implementation scope of NVIDIA-NeMo#3504 after review history is preserved. ## Consolidated stack links ### Gym 1. [Gym NVIDIA-NeMo#3999 — SA-V tracking verifier](NVIDIA-NeMo#3999) 2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout identity](NVIDIA-NeMo#4000) ### NeMo-RL 1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model loading](NVIDIA-NeMo/RL#4472) 2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame contract](NVIDIA-NeMo/RL#4473) 3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing runtime](NVIDIA-NeMo/RL#4474) 4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI fixtures](NVIDIA-NeMo/RL#4475) 5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency pins](NVIDIA-NeMo/RL#4476) ### Child dependency refs - [MCore integration ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004), adapting [MCore #7278](NVIDIA/Megatron-LM#7278) and [MCore #7279](NVIDIA/Megatron-LM#7279) to the pinned base. - [MBridge integration ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004). --------- Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com> Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The terminal HTTP 400 scenario currently excludes verifier evidence and skips the verifier outcome/reward checks. A harness can therefore pass while skipping grading or masking a graded rollout.
Require verifier evidence for this scenario, reuse the existing observed-outcome and saved-reward checks, and add one P0 behavioral check requiring
evaluation_completed=trueandmask_sample=false. Final-answer checking remains inapplicable when no answer was generated. The same saved-outcome check applies to the other controlled scenarios, whose verifier also completes normally.Follow-up to #4450. No separate GitHub issue is needed for this focused gap in the existing scenario. No producer or health-engine changes; stricter conformance can expose previously hidden producer failures. Keep draft pending the Hermes masking follow-up.
Validation:
python -m pytest -o addopts= --import-mode=importlib -q scripts/harness_conformance/tests tests/unit_tests/harness_capabilities tests/unit_tests/test_harness_conformance_checks.py tests/unit_tests/test_harness_conformance_ci.py: 379 passed, 2 skipped.pre-commit run --all-files: passed.Fresh local scripted conformance on this change:
opencode_agentcodex_agentpi_agenthermes_agentmodel_error:verifier.saved_outcomeHermes independently calls the verifier and saves reward
0.0andevaluation_completed=true, then savesmask_sample=true. This is the previously hidden producer gap exposed by the new check. No expectation is weakened to accept it. Pi was rerun from scratch with its installed 0.80.2 binary on PATH after the first launch could not find it.Regression coverage: skipped verifier, changed reward, masked/incomplete result, missing completion flag, and missing rollout delivery.
External live-model smoke: N/A; this changes the conformance checker only. The scripted suite executes the real harness processes and collects fresh rollouts.
GitHub lint, copyright, secrets and DCO checks passed. No inline review comments at inspection. Full conformance CI is not automatically dispatched for a draft PR; the copy-PR bot reports that restriction.