Skip to content

fix(tb4): upgrade to 4.1 and preserve artifacts and limit outcomes - #6220

Merged
lafrancef merged 2 commits into
mainfrom
flafrance/fer-421-upstream-tb4-fixes-to-external-gym
Nov 16, 2026
Merged

lafrancef merged 2 commits into
mainfrom
flafrance/fer-421-upstream-tb4-fixes-to-external-gym

Conversation

@lafrancef

@lafrancef lafrancef commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Updates both TB4 profiles to the official Terminal-Bench 4.1 task and Compose-image pins.

  • Exclude the shared S3 mount from TB4 agent and verifier sandboxes, including the Harbor deployment overlay.
  • Keep safe artifact entries when another entry is unsafe, record refused entries, and apply the same filtering to shared-log snapshots.
  • Recover missing-reward verifier logs within the verifier deadline.
  • Recognize full and truncated pod storage-limit death reports. Preserve their completed zero through cleanup, strict collection and recovery, keeping cleanup failure events as observations. Unrelated sandbox losses retain infrastructure-failure handling.

Terminal-Bench 4.1 changes the verifiers, so scores and partial_credit are not comparable with 4.0. Artifact filtering can also change grades when an artifact contains unsafe entries. Existing Harbor datasets must be refreshed using the entry README's version-migration steps. Separate GPU provider blocks need the documented shared-mount opt-out label too.

Live rollout validation remains deferred because the validation host could not reach the required services.

No separate public issue; this updates the existing TB4 integration and preserves the failure-handling contracts from #5194 and #6114.

This consolidates the storage-limit fix also proposed in #6219.

Update the official task pins across Harbor and relay profiles. Retain safe
artifacts and bounded missing-reward logs, and keep confirmed sandbox
allocation-limit deaths settled as zero through collection and recovery.

Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@lafrancef lafrancef added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working labels Nov 16, 2026
@lafrancef
lafrancef marked this pull request as ready for review November 16, 2026 10:48
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
@lafrancef
lafrancef requested a review from a team as a code owner November 16, 2026 10:54
@lafrancef

Copy link
Copy Markdown
Contributor Author

/ok to test 816f7dd

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 16, 2026 10:55
@github-actions

Copy link
Copy Markdown
Contributor

@lafrancef

Copy link
Copy Markdown
Contributor Author

@wprazuch, could you review and approve this? It includes the storage-limit fix from #6219 alongside the TB4.1 update and sandbox/artifact fixes.

Core, sandbox, all eight server shards, and the combined coverage gate passed for 816f7dd3077e3f41da620804301e20ae1a17dbfe (CI run). The remaining non-blocking compliance finding flags the existing legacy run() in responses_api_agents/miniswe_sandboxed_agent/app.py, which this PR does not change.

@lafrancef
lafrancef added this pull request to the merge queue Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43875001646

Merged via the queue into main with commit bc38f7b Nov 16, 2026
65 of 67 checks passed
@lafrancef
lafrancef deleted the flafrance/fer-421-upstream-tb4-fixes-to-external-gym branch November 16, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants