Skip to content

test(conformance): require grading after terminal HTTP 400 - #4473

Merged
lafrancef merged 1 commit into
mainfrom
codex/http400-verifier-conformance
Oct 12, 2026
Merged

lafrancef merged 1 commit into
mainfrom
codex/http400-verifier-conformance

Conversation

@giuliolovisotto

@giuliolovisotto giuliolovisotto commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

The terminal HTTP 400 scenario currently excludes verifier evidence and skips the verifier outcome/reward checks. A harness can therefore pass while skipping grading or masking a graded rollout.

Require verifier evidence for this scenario, reuse the existing observed-outcome and saved-reward checks, and add one P0 behavioral check requiring evaluation_completed=true and mask_sample=false. Final-answer checking remains inapplicable when no answer was generated. The same saved-outcome check applies to the other controlled scenarios, whose verifier also completes normally.

Follow-up to #4450. No separate GitHub issue is needed for this focused gap in the existing scenario. No producer or health-engine changes; stricter conformance can expose previously hidden producer failures. Keep draft pending the Hermes masking follow-up.

Validation:

  • python -m pytest -o addopts= --import-mode=importlib -q scripts/harness_conformance/tests tests/unit_tests/harness_capabilities tests/unit_tests/test_harness_conformance_checks.py tests/unit_tests/test_harness_conformance_ci.py: 379 passed, 2 skipped.

  • pre-commit run --all-files: passed.

  • Fresh local scripted conformance on this change:

    Harness Passing scenarios Remaining failure
    opencode_agent 7/7 None
    codex_agent 7/7 None
    pi_agent 7/7 None
    hermes_agent 6/7 model_error: verifier.saved_outcome

    Hermes independently calls the verifier and saves reward 0.0 and evaluation_completed=true, then saves mask_sample=true. This is the previously hidden producer gap exposed by the new check. No expectation is weakened to accept it. Pi was rerun from scratch with its installed 0.80.2 binary on PATH after the first launch could not find it.

  • Regression coverage: skipped verifier, changed reward, masked/incomplete result, missing completion flag, and missing rollout delivery.

  • External live-model smoke: N/A; this changes the conformance checker only. The scripted suite executes the real harness processes and collects fresh rollouts.

GitHub lint, copyright, secrets and DCO checks passed. No inline review comments at inspection. Full conformance CI is not automatically dispatched for a draft PR; the copy-PR bot reports that restriction.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
@giuliolovisotto giuliolovisotto added bug Something isn't working area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters labels Oct 12, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@giuliolovisotto
giuliolovisotto marked this pull request as ready for review October 12, 2026 14:35
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 12, 2026 14:35
@yaoyu-33 yaoyu-33 added complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 12, 2026
@lafrancef
lafrancef added this pull request to the merge queue Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38285005417

Merged via the queue into main with commit eed77e1 Oct 12, 2026
52 of 54 checks passed
@lafrancef
lafrancef deleted the codex/http400-verifier-conformance branch October 12, 2026 17:49
adil-a added a commit that referenced this pull request Oct 13, 2026
A model endpoint that refuses the request for what it asked (HTTP 400, 413 or 422 after
Hermes's own retries) is a model outcome: the episode reaches the verifier and keeps an
unmasked zero, as the conformance model_error scenario requires since #4473. Auth, routing,
rate-limit and server errors stay provider failures and are masked as #4048 intended. The
response records the status under metadata.model_rejected.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 15, 2026
## Summary
- Add the SA-V tracking verifier and its server configuration.
- Declare the SA-V environment servers used by video RL.
- Add portable five-row example data, schema, metrics, and checked
rollout artifacts.

## Why
This is the Gym-side verifier foundation for mixed SA-V tracking and
CapRL video training in NeMo-RL. No separate issue is needed; this is
part of the Super 3.5 video RL upstreaming stack.

## Validation
- Runtime evidence: the [healthy 16-node async `r18` W&B
run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003)
shows repeated GRPO cycles, stable TMPE, and increasing reward on the
equivalent video implementation.
- `gym dataset collate ... +mode=example_validation`: 5 examples;
generated `data/example_metrics.json`.
- Focused validation: 25 verifier tests and all 5 example rows against
`task_data.py` passed.
- Ruff format and lint passed for the changed Python files.
- Both Gym stack ranges pass `git diff --check`.

## Rollouts
`data/example_rollouts.jsonl` contains one checked rollout for each of
the five example rows. Representative task checks:

- `pt`:
`<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, visible-frame mask membership `true`.
- `box`:
`<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, frame IoU `1.0`.
- `ref`:
`<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` —
`format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT
object and mask membership `true`.

The artifact also contains visible/invisible `pt` coverage and an
all-absent `ref` negative case, preserving the verifier's current
unmatched-ID behavior.

## Compatibility
Adds a new verifier/configuration and does not change unrelated
environments. Example media is embedded as a small inline JPEG, so no
employee or cluster-local paths are required.

## Stack
Gym PR 1 of 2. Consolidates and supersedes the implementation scope of
NVIDIA-NeMo#3504 after review history is preserved.

## Consolidated stack links

### Gym
1. [Gym NVIDIA-NeMo#3999 — SA-V tracking
verifier](NVIDIA-NeMo#3999)
2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout
identity](NVIDIA-NeMo#4000)

### NeMo-RL
1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model
loading](NVIDIA-NeMo/RL#4472)
2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame
contract](NVIDIA-NeMo/RL#4473)
3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing
runtime](NVIDIA-NeMo/RL#4474)
4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI
fixtures](NVIDIA-NeMo/RL#4475)
5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency
pins](NVIDIA-NeMo/RL#4476)

### Child dependency refs
- [MCore integration
ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004),
adapting [MCore #7278](NVIDIA/Megatron-LM#7278)
and [MCore #7279](NVIDIA/Megatron-LM#7279) to
the pinned base.
- [MBridge integration
ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004).

---------

Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 69ecd231 Deployed Oct 12, 2026 by copy-pr-bot[bot] via release / finalize / notify #5034
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants