Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 55 additions & 4 deletions .github/workflows/config/.secrets.baseline

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions ATTRIBUTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@ All third-party software is used without modification unless otherwise noted

---

## Benchmark Data and Prompts

- FACTS Grounding Public Examples, Google DeepMind and Google Research, Kaggle dataset
deepmind/FACTS-grounding-examples, version 17: Apache-2.0. The full dataset is downloaded
by the benchmark prepare script and is not committed.
- FACTS Grounding v2 technical report: Cheng et al., The FACTS Leaderboard: A Comprehensive
Benchmark for Large Language Model Factuality, arXiv:2512.10791v1, section 6 (CC BY 4.0).
- The official Kaggle starter notebook, prathameshbang/facts-grounding-v2-benchmark-starter,
version 4 (Apache-2.0), is the reference for the eligibility and grounding prompt/parser
behavior. The adapter port is version-pinned by the resources-server prompt hashes.

## Python Package Dependencies

| Component | Version | License | URL |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Equivalence Rule | knowledge | Question - Answering with rule-based reward | Improve retrieval and counting capabilities | - | - | - | <a href='resources_servers/equivalence_rule/configs/lc.yaml'>lc.yaml</a> | - |
| Ether0 | knowledge | ether0 chemistry benchmark verifiers | Evaluate chemistry knowledge and reasoning with ether0 benchmark | - | ✓ | - | <a href='resources_servers/ether0/configs/ether0.yaml'>ether0.yaml</a> | - |
| Evalplus | coding | Function-completion code judged by EvalPlus base + plus tests (HumanEval+, MBPP+) | Improve Python function-completion capabilities | - | - | - | <a href='resources_servers/evalplus/configs/evalplus.yaml'>evalplus.yaml</a> | - |
| Facts Grounding V2 | long_context | FACTS Grounding v2 long-form grounded generation judged for eligibility and sentence-level grounding by Gemini 2.5 Flash and GPT-5 | Measure whether long-form answers stay grounded in a supplied document and still address the request | - | - | - | <a href='resources_servers/facts_grounding_v2/configs/facts_grounding_v2.yaml'>facts_grounding_v2.yaml</a> | - |
| False Statement Judge | math | Sycophancy grader for prove-the-false-statement benchmarks, using MathArena's 0-2 judge rubric | Measure how often a model claims to prove a statement that is false as written | - | - | - | <a href='resources_servers/false_statement_judge/configs/false_statement_judge.yaml'>false_statement_judge.yaml</a> | - |
| Finance Agent V2 | agent | Vals finance-agent-v2 tools (web/EDGAR/price/calculator) for financial research questions | Run the official Vals FABv2 agent tools at scale in NeMo Gym | - | - | - | <a href='resources_servers/finance_agent_v2/configs/finance_agent_v2.yaml'>finance_agent_v2.yaml</a> | - |
| Finance Sec Search | agent | SEC EDGAR filing search for financial analysis questions | Enable LLMs to search and analyze SEC filings | - | - | - | <a href='resources_servers/finance_sec_search/configs/finance_sec_search.yaml'>finance_sec_search.yaml</a> | - |
Expand Down
71 changes: 71 additions & 0 deletions benchmarks/facts_grounding_v2/METRICS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# FACTS Grounding v2 metrics and protocol

## Scope

FACTS Grounding measures whether a model can produce a long-form answer using only a supplied
document while still addressing the user's request. This adapter implements the official v2
judge protocol for the 856-row public Kaggle release. It does not include the private holdout
set or submit results to the Kaggle leaderboard.

The source of record is Kaggle's FACTS Grounding Public Examples dataset, version 17. The
prepare script pins and verifies the public source files before producing NeMo Gym rows.
The December 2024 Hugging Face mirror has 860 rows; four rows are absent from the pinned
public release and are not scored.

## Judging protocol

The benchmark keeps eligibility and groundedness separate:

1. Gemini 2.5 Flash, then GPT-5, independently writes a baseline answer and rates whether the
policy response follows the request. The first well-formed rating other than Major Issue(s)
determines eligibility. An unparseable eligibility rating follows the official protocol
and is treated as eligible.
2. Each judge labels the response sentence by sentence for support from the supplied context.
A judge's answer is grounded when at least one sentence is parsed and none is
not_supported. A response with no parseable grounding sentence is not grounded.
3. The adjusted factuality reward is zero for an ineligible response; otherwise it is the mean
of the two judges' grounded/not-grounded verdicts. The unadjusted groundedness score is
retained separately. Both judges run even for ineligible responses, matching the v2
reference protocol.

## Metric dictionary

| Metric | Definition | Denominator / notes |
|---|---|---|
| factuality_score | Mean adjusted reward; the primary public-set score | All scored answers; report judge-failed rows separately |
| unadjusted_factuality_score | Mean grounding verdict with ineligible answers included | All answers with grounding verdicts |
| eligibility_rate | Fraction of answers judged eligible | All answers |
| grounded_rate_eligible/<judge> | Eligible answers that judge found fully supported | Eligible answers |
| grounded_all_judges_rate_eligible | Eligible answers grounded by both judges | Eligible answers |
| judge_disagreement_rate_eligible | Eligible answers with exactly one grounded verdict | Eligible answers |
| eligibility_decided_by/<judge> | Fraction whose eligibility was settled by that judge | All answers |
| eligibility_invalid_rate | Fraction with an unparseable eligibility rating | All answers |
| grounding_parse_empty_rate/<judge> | Fraction with no parsed sentence-level verdict | Answers graded by that judge |
| grounding_unparseable_line_rate/<judge> | Fraction with at least one unparseable sentence line | Answers graded by that judge |
| factuality_score_ci95_low/high | Bootstrap 95% interval for mean adjusted reward | Fixed seed and 2,000 resamples |
| generation_empty_rate, generation_truncated_rate | Empty and output-cap-truncated generations | All answers |
| factuality_score/<slice>/<value> | Adjusted score for a domain, high-level type, or task-type slice | Rows in that slice |

Judge transport failures are routed to the failure sidecar and excluded from the score until
retried. Unparseable judge content is not a transport failure: it follows the official
eligibility or grounding parsing rules and is reflected in the corresponding diagnostics.

## Row mapping

| Source field or operation | NeMo Gym representation |
|---|---|
| Kaggle full_prompt | One user message in responses_create_params.input; no prompt template is applied |
| user_request and context_document | Task data consumed by the eligibility and grounding judges |
| system_instruction, domain, type, high_level_type | Preserved task data and result-slice metadata |
| Official eligibility decision | eligibility_ratings, eligible, eligibility_deciding_judge |
| Sentence-level grounding decisions | judge_receipts, grounding_verdicts, grounding_label_counts, parse diagnostics |
| Official reward and component score | reward, grounding_score, unadjusted_score |

## Interpretation

- Report the public-set denominator and the two judge identities with every factuality score.
- A score of 0.5 for one eligible answer means the two judges disagreed on groundedness.
- Do not compare this public-only score directly with Kaggle leaderboard scores, which combine
public and private examples and may use a different execution pipeline.
- Empty, truncated, missing, duplicate, and judge-failed rows are separate outcomes; do not
silently remove them from the denominator or describe infrastructure failures as model errors.
11 changes: 11 additions & 0 deletions benchmarks/facts_grounding_v2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# FACTS Grounding v2

This benchmark adapter prepares the 856-row public FACTS Grounding release and implements
the v2 eligibility and sentence-level grounding protocol. The private Kaggle split is not
included. The prepare script downloads the pinned public release and verifies its source
files before producing NeMo Gym JSONL.

- Environment configuration and run instructions:
[resources_servers/facts_grounding_v2](../../resources_servers/facts_grounding_v2/README.md)
- Metric definitions and interpretation: [METRICS.md](METRICS.md)
- Source report and benchmark references: [paper/PAPER.md](paper/PAPER.md)
2 changes: 2 additions & 0 deletions benchmarks/facts_grounding_v2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
17 changes: 17 additions & 0 deletions benchmarks/facts_grounding_v2/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
config_paths:
- resources_servers/facts_grounding_v2/configs/facts_grounding_v2.yaml

facts_grounding_v2_public:
_inherit_from: facts_grounding_v2_simple_agent
responses_api_agents:
simple_agent:
datasets:
- name: facts_grounding_v2
type: benchmark
jsonl_fpath: benchmarks/facts_grounding_v2/data/facts_grounding_v2_public.jsonl
prepare_script: benchmarks/facts_grounding_v2/prepare.py
# Rows carry `full_prompt` as the single user message, exactly as the official starter prompts the
# evaluated model; no prompt template is applied.
prompt_config: null
num_repeats: 1
license: Apache 2.0
2 changes: 2 additions & 0 deletions benchmarks/facts_grounding_v2/data/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*
!.gitignore
1 change: 1 addition & 0 deletions benchmarks/facts_grounding_v2/paper/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.pdf
28 changes: 28 additions & 0 deletions benchmarks/facts_grounding_v2/paper/PAPER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Benchmark technical reports

## FACTS Grounding v2 (the protocol this adapter implements)

- Title: The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
- Authors: Aileen Cheng, Alon Jacovi, Amir Globerson, Ben Golan, Charles Kwong, Chris Alberti, Connie Tao, Eyal Ben-David, Gaurav Singh Tomar, Lukas Haas, Yonatan Bitton, et al. (Google DeepMind, Google Research, Kaggle)
- Canonical source: https://arxiv.org/abs/2512.10791v1 (version v1, 11 December 2025)
- Immutable PDF: https://arxiv.org/pdf/2512.10791v1
- SHA-256 of the PDF: `db046e76cc1877880843d0e7fd4898422f1064d47f8990b04f3c230229ede6be`
- Mirror published by the authors: https://storage.googleapis.com/deepmind-media/FACTS/FACTS_benchmark_suite_paper.pdf
(SHA-256 `f079762c7a3ce79e725e79fa009fa3afd63eb3cbf77f6507c1f9901640752cb7`)
- License: CC BY 4.0
- Leaderboard: https://www.kaggle.com/benchmarks/google/facts-grounding
- Official public data: https://www.kaggle.com/datasets/deepmind/FACTS-grounding-examples (version 17)
- Official reference implementation: Kaggle starter notebook `prathameshbang/facts-grounding-v2-benchmark-starter` (version 4)

## FACTS Grounding v1 (prompt collection and the original judge design)

- Title: The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
- Authors: Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, et al.
- Canonical source: https://arxiv.org/abs/2501.03200
- Public data mirror: https://huggingface.co/datasets/google/FACTS-grounding-public (revision `11b6961370aa0ac73c91d58e35317e724b2ac765`, 860 rows)

The PDF is not committed to keep binaries out of the repository. `python benchmarks/facts_grounding_v2/fetch_paper.py`
downloads the immutable arXiv version of the suite report into this directory (gitignored) and verifies the
SHA-256; run packages copy the verified file under `paper/`. Section 6 of the suite report defines the v2 protocol:
unchanged prompts, Gemini 2.5 Flash and GPT-5 as judges with the v2 prompt, an unadjusted factuality score averaged
over judges, and disqualification of ineligible responses.
Loading