Repository navigation
feat(facts-grounding-v2): add public benchmark adapter - #3661
Draft
reinainblood wants to merge 2 commits into
Draft
reinainblood wants to merge 2 commits into
reinainblood wants to merge 2 commits into
Conversation
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
Signed-off-by: Kirsten Ruge <kirstenruge@gmail.com>
This was referenced Oct 22, 2026
yaoyu-33
marked this pull request as draft
November 18, 2026 02:18
Contributor
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds FACTS Grounding v2 as a NeMo Gym benchmark and resources server. The adapter prepares the pinned 856-prompt public set, sends each official full prompt as one user message, and implements the v2 eligibility and sentence-level grounding protocol with Gemini 2.5 Flash followed by GPT-5. It includes a TaskData schema, five example inputs and real example rollouts, tests, metric definitions, attribution, and catalog documentation. The private holdout set is not included.
No issue is linked; this is a self-contained integration of the published FACTS Benchmark Suite.
Contributor: Kirsten Ruge, Snorkel AI.
The source is Kaggle
deepmind/FACTS-grounding-examplesversion 17. The source CSV SHA-256 is66990aea86a91d40edcda6ebcd0d686674246dc7c07338ed8c17e1c275b1e096;gym eval prepareproduces 856 Gym rows with JSONL SHA-256191e6674a8ba007796e3227487442c4ae0fcb44c2d95a54bd42a7b36679a1636.Validation
Final branch synchronization (2026-10-01): merged upstream
mainat31179922;pre-commit run --all-filesandgit diff --checkpassed on the resulting tree. GitHub DCO passed on the pushed head. Full NVIDIA CI is awaiting external-contributor vetting.uv run gym eval prepare --benchmark facts_grounding_v2: 856 rows; source and prepared-data hashes verified.uv run gym dataset collate --resources-server facts_grounding_v2 --mode example_validation --output-dir <temporary-directory>: five example rows; Gym-generated example metrics.uv run gym dataset collate --config benchmarks/facts_grounding_v2/config.yaml --mode train_preparation --output-dir <temporary-directory>: 856 benchmark rows with 856 unique IDs and the public agent astask_source.uv run ng_test +entrypoint=resources_servers/facts_grounding_v2 +should_validate_data=true: 23 tests and data validation passed.uv run pytest resources_servers/facts_grounding_v2/tests: 23 passed.uv run pytest tests/unit_tests/test_task_data.py: 331 passed.gym list benchmarks facts_grounding_v2: benchmark discovered.pre-commit run --all-files: passed.Full core and sandbox suites were also compared locally against pristine
upstream/main. The macOS Bash 3.2 shell-array failures reproduced on both branches; the sandbox suite had 862 passed, one Enroot/procfailure, and three skipped on both. A targeted comparison found no PR-only failed node. The public SHA-256 entries added to the secrets baseline cause full CI classification, so Linux CI is still required.Rollouts
Each model completed one rollout on each of the same 856 public task IDs, with no missing or duplicate scored IDs. Sampling was temperature 0, top-p 1, and 16,384 maximum output tokens. The judges were Gemini 2.5 Flash and GPT-5 in the adapter's specified order. Scores are adjusted factuality with Gym 95% bootstrap intervals.
moonshotai/Kimi-K3Qwen/Qwen3.5-122B-A10B-FP8nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16The Qwen and Super empty outputs remain in the denominator as model output outcomes. Super's four earlier judge-error attempts remain in the failure sidecar; each affected task later received a scored row. Full run artifacts and reports will be delivered separately from the source PR. The repository includes five sanitized Kimi example rollouts to exercise the validation contract.
Compatibility and benchmark impact
This is additive: it registers the public benchmark and its scoring resources server without changing existing benchmark behavior. It does not include the private holdout or the four full rollout files in the source repository. Metric definitions and protocol differences from the original public examples are documented in
benchmarks/facts_grounding_v2/METRICS.md.Checklist
pre-commit run --all-files).git commit -s).