Skip to content

feat(facts-grounding-v2): add public benchmark adapter - #3661

Draft
reinainblood wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
reinainblood:codex/facts-grounding-v2-upstream-clean
Draft

reinainblood wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
reinainblood:codex/facts-grounding-v2-upstream-clean

Conversation

@reinainblood

@reinainblood reinainblood commented Sep 23, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds FACTS Grounding v2 as a NeMo Gym benchmark and resources server. The adapter prepares the pinned 856-prompt public set, sends each official full prompt as one user message, and implements the v2 eligibility and sentence-level grounding protocol with Gemini 2.5 Flash followed by GPT-5. It includes a TaskData schema, five example inputs and real example rollouts, tests, metric definitions, attribution, and catalog documentation. The private holdout set is not included.

No issue is linked; this is a self-contained integration of the published FACTS Benchmark Suite.

Contributor: Kirsten Ruge, Snorkel AI.

The source is Kaggle deepmind/FACTS-grounding-examples version 17. The source CSV SHA-256 is 66990aea86a91d40edcda6ebcd0d686674246dc7c07338ed8c17e1c275b1e096; gym eval prepare produces 856 Gym rows with JSONL SHA-256 191e6674a8ba007796e3227487442c4ae0fcb44c2d95a54bd42a7b36679a1636.

Validation

  • Final branch synchronization (2026-10-01): merged upstream main at 31179922; pre-commit run --all-files and git diff --check passed on the resulting tree. GitHub DCO passed on the pushed head. Full NVIDIA CI is awaiting external-contributor vetting.

  • uv run gym eval prepare --benchmark facts_grounding_v2: 856 rows; source and prepared-data hashes verified.

  • uv run gym dataset collate --resources-server facts_grounding_v2 --mode example_validation --output-dir <temporary-directory>: five example rows; Gym-generated example metrics.

  • uv run gym dataset collate --config benchmarks/facts_grounding_v2/config.yaml --mode train_preparation --output-dir <temporary-directory>: 856 benchmark rows with 856 unique IDs and the public agent as task_source.

  • uv run ng_test +entrypoint=resources_servers/facts_grounding_v2 +should_validate_data=true: 23 tests and data validation passed.

  • uv run pytest resources_servers/facts_grounding_v2/tests: 23 passed.

  • uv run pytest tests/unit_tests/test_task_data.py: 331 passed.

  • gym list benchmarks facts_grounding_v2: benchmark discovered.

  • pre-commit run --all-files: passed.

Full core and sandbox suites were also compared locally against pristine upstream/main. The macOS Bash 3.2 shell-array failures reproduced on both branches; the sandbox suite had 862 passed, one Enroot /proc failure, and three skipped on both. A targeted comparison found no PR-only failed node. The public SHA-256 entries added to the secrets baseline cause full CI classification, so Linux CI is still required.

Rollouts

Each model completed one rollout on each of the same 856 public task IDs, with no missing or duplicate scored IDs. Sampling was temperature 0, top-p 1, and 16,384 maximum output tokens. The judges were Gemini 2.5 Flash and GPT-5 in the adapter's specified order. Scores are adjusted factuality with Gym 95% bootstrap intervals.

Model Adjusted factuality (95% CI) Unadjusted Eligibility Quality accounting
moonshotai/Kimi-K3 66.47% (64.14–68.93) 66.76% 99.42% 856 healthy; no empty or truncated generations
Qwen/Qwen3.5-122B-A10B-FP8 71.79% (69.33–74.07) 72.78% 98.48% 848 healthy; eight empty, truncated outputs scored as zero
nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 51.99% (49.30–54.73) 52.63% 98.83% 856 healthy; one truncated but nonempty generation
nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 72.08% (69.74–74.30) 72.55% 99.18% 853 healthy; three empty, truncated outputs scored as zero

The Qwen and Super empty outputs remain in the denominator as model output outcomes. Super's four earlier judge-error attempts remain in the failure sidecar; each affected task later received a scored row. Full run artifacts and reports will be delivered separately from the source PR. The repository includes five sanitized Kimi example rollouts to exercise the validation contract.

Compatibility and benchmark impact

This is additive: it registers the public benchmark and its scoring resources server without changing existing benchmark behavior. It does not include the private holdout or the four full rollout files in the source repository. Metric definitions and protocol differences from the original public examples are documented in benchmarks/facts_grounding_v2/METRICS.md.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated edits are tracked separately.
  • Tests added or updated and pass locally; platform-dependent full-suite limits are described above.
  • Documentation added or updated.
  • Pre-commit checks pass locally (pre-commit run --all-files).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s).

Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
@reinainblood
reinainblood requested a review from a team as a code owner September 23, 2026 14:58
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Sep 23, 2026
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 23, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 24, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 25, 2026
@yaoyu-33 yaoyu-33 removed the needs-review PR is ready for code review and waiting on a reviewer label Sep 25, 2026
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the benchmark-onboarding Self-contained benchmark onboarding label Oct 21, 2026
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 02:18
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers benchmark-onboarding Self-contained benchmark onboarding community-request Issue reported or requested by someone from the community complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants