Skip to content

🧪 evals: assert on what an agent did, not on what it said - #40

Merged
Jaweria-B merged 2 commits into
mainfrom
evals/agent-grounding-harness
Jul 26, 2026
Merged

Jaweria-B merged 2 commits into
mainfrom
evals/agent-grounding-harness

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Jul 26, 2026 •

Copy link
Copy Markdown
Collaborator

Every grounding defect this project has hit was found by a human running the CLI and noticing the answer contained a number nobody had retrieved — the false-healthy RAG status (#37), the weak-hit answering and the execution agent routing past the docs (#39). None of them was catchable by the existing suite, which exercises tools in isolation while the defects live in the agent's behaviour: which tool it reached for, and whether its prose is traceable to what that tool returned.

CONTRIBUTING.md already claimed these boundaries were "validated by the eval harness". There was no eval harness. This adds one.

What it asserts

A run is recorded into a RunTrace — tool calls in order, each call's output, and the final answer — via capture_run_messages. Three checks, each corresponding to a failure that actually shipped:

  • assert_consulted_docs — did it look anything up before answering? The 🐛 rag: stop the agents answering from memory when the docs don't cover it #39 routing defect, expressed as an assertion instead of something a human has to notice.
  • assert_grounded_quantities — does every physical quantity in the answer appear in some tool output? Only numbers carrying a unit or bound to a named simulation parameter count, so "step 2 of 3" isn't mistaken for a physics claim; and values are normalised, so 60 in the answer matches 60.0 in a tool return rather than being reported as invented. Without both filters the check cries wolf and stops being believed.
  • assert_cited — an answer built on retrieved docs carrying no citation is a model that retrieved and then answered from memory anyway.

Grounding is checked against all tool output, not just the documentation search: a k-point spacing taken from build_workflow_inputs is grounded just as legitimately as one quoted from the docs.

Two tiers

tests/evals/test_harness.py — runs in CI, offline, deterministic. It tests the assertions themselves. An eval that quietly degrades into always-passing is worse than none, because it certifies nothing while looking green. So it scripts a fabricating agent with FunctionModel — searches, gets a real excerpt, then states a spacing that appears nowhere in it — and requires the check to catch that, then requires a correctly-quoted answer to pass. Same for skipped retrieval and missing citations.

tests/evals/test_grounding.py — opt-in (AIIDA_AGENTS_EVAL=1, marked llm). Needs a real model and a built index, so it never runs in CI. Covers both directions of the #39 routing fix: knowledge questions must reach the docs, and setup requests must still discover entry points rather than detouring into them — splitting the two apart could easily over-correct, and nothing else would catch that.

The opt-in tier is flaky by nature. A weaker model fails it more often, and that is a result rather than a broken test — the module docstring says so explicitly, so nobody later "fixes" a failure by loosening an assertion.

Test plan

  • ruff / ruff format / mypy / uv-lock clean via pre-commit
  • Full suite: 514 passed, 12 skipped on this branch rebased onto current main
  • The 10 new skips are the opt-in tier declining without AIIDA_AGENTS_EVAL=1; the llm marker is registered in pyproject.toml, so no unknown-marker warning
  • Not yet run against a real model — that needs a configured provider and a built index. Worth doing before trusting the results; some cases may well fail on a free-tier model, which is the intended signal rather than a defect in the harness.

CONTRIBUTING.md gains a section on when to run it: after changing a system prompt, a tool description, or anything in src/aiida_agents/rag/.

Every grounding defect this project has hit was found by a human running the
CLI and noticing the answer contained a number nobody had retrieved: the
false-healthy RAG status, the weak-hit answering, the execution agent routing
past the docs. None of them was catchable by the existing suite, which
exercises tools in isolation while the defects live in the agent's behaviour --
which tool it reached for, and whether its prose is traceable to what that tool
returned. CONTRIBUTING already claimed these boundaries were "validated by the
eval harness"; there was no eval harness.

Add tests/evals. A run is recorded into a RunTrace (tool calls in order, each
one's output, the final answer) via capture_run_messages, and asserted on:

- assert_consulted_docs -- did it look anything up before answering, the
  routing defect expressed as an assertion.
- assert_grounded_quantities -- does every physical quantity in the answer
  appear in some tool output. Only numbers carrying a unit or bound to a named
  simulation parameter count, so "step 2 of 3" is not mistaken for a physics
  claim, and 60 in the answer matches 60.0 in a tool return rather than being
  reported as invented.
- assert_cited -- an answer built on retrieved docs that carries no citation is
  a model that retrieved and then answered from memory anyway.

Two tiers. test_harness.py runs in CI on FunctionModel: it scripts a
fabricating agent and requires each check to catch it, then scripts the correct
behaviour and requires each to pass, so the assertions cannot rot into
always-passing. test_grounding.py is opt-in (AIIDA_AGENTS_EVAL=1, marked llm),
needs a real model and a built index, and covers both directions of the routing
fix -- knowledge questions must reach the docs, and setup requests must still
discover entry points rather than detouring into them.
@Jaweria-B
Jaweria-B requested a review from GeigerJ2 July 26, 2026 14:13
@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a grounding evaluation harness that records agent tool traces, checks evidence and citations, validates behavior with deterministic tests, and runs opt-in real-model evaluations. Documentation and pytest configuration describe the command, marker, environment, and evaluation tiers.

Changes

Grounding evaluations

Layer / File(s) Summary
Trace capture and grounding assertions
tests/evals/_harness.py
Adds trace structures, tool-output pairing, environment helpers, quantity grounding checks, documentation consultation checks, and citation validation.
Deterministic harness validation
tests/evals/test_harness.py
Tests trace capture, fabricated quantities, retrieval and citation enforcement, numeric normalization, and .env configuration loading with scripted agents.
Opt-in real-model grounding evaluations
tests/evals/test_grounding.py
Adds gated analysis and execution agent evaluations for documentation retrieval, citations, grounded quantities, setup discovery, and missing-index reporting.
Evaluation commands and pytest configuration
pyproject.toml, CONTRIBUTING.md
Registers the llm marker and documents commands, environment requirements, evaluation tiers, model configuration, and expected flakiness.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EvalTest
  participant Agent
  participant SearchTool
  participant RunTrace
  participant Assertions
  EvalTest->>Agent: run prompt
  Agent->>SearchTool: search documentation
  SearchTool-->>Agent: return evidence
  Agent-->>RunTrace: record tool calls and answer
  RunTrace->>Assertions: validate quantities, retrieval, and citations
  Assertions-->>EvalTest: pass or fail evaluation
Loading

Suggested reviewers: geigerj2

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding evals that judge agent behavior and grounding rather than output text alone.
Description check ✅ Passed The description is directly about the added evaluation harness, its checks, and the new opt-in grounding tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jaweria-B
Jaweria-B removed the request for review from GeigerJ2 July 26, 2026 14:19
The first real-model run of the opt-in tier failed all ten cases without
producing any grounding signal: every request went to a local Ollama endpoint
instead of the configured provider. tests/conftest.py chdirs every test into an
empty temp directory -- deliberately, so a developer's .env cannot leak into
hermetic tests -- and pydantic-settings resolves .env relative to the working
directory. The eval fixtures inherited that, found no .env, and fell back to
ModelSettings' bare defaults (ollama/qwen3.5:2b), which then exhausted its
context without answering. Ten failures for a model that was never contacted.

Copy the project's .env into the test's cwd, which is what the conftest
docstring already prescribes for a test that needs one. The fixture depends on
_isolate_cwd by name rather than trusting fixture ordering, since the copy has
to land after the chdir or it goes to the repository root. Exported shell
variables survive the chdir and are unaffected; they still take precedence, as
pydantic-settings intends.

Also log the resolved provider/model on every case, passing or failing. The
underlying problem was not the fallback but its invisibility: a grounding
result is meaningless without knowing which model produced it, and reading ten
failures as model weakness when the model was never reached is exactly the
wrong conclusion to invite.

Tested in CI rather than only in the opt-in tier: the new cases assert the bare
defaults are what an empty cwd yields, then that the copy makes the configured
values win -- the bug and the fix pinned in one test.
@Jaweria-B
Jaweria-B merged commit d4d1dbd into main Jul 26, 2026
4 of 5 checks passed
@Jaweria-B
Jaweria-B deleted the evals/agent-grounding-harness branch July 26, 2026 16:12

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/evals/test_grounding.py`:
- Around line 147-162: Update the execution-agent documentation-answer tests
around test_no_invented_physics_values and the preceding parameterized case so
both answer paths require a user-facing citation. Ensure the numeric
cutoff/spacing case continues to validate grounded quantities and additionally
asserts that documentation was consulted, while the other documentation-derived
case also asserts citation presence.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ef1fce60-d1d0-4e1b-95fb-9cd379563e5e

📥 Commits

Reviewing files that changed from the base of the PR and between fc5260e and e2f136a.

📒 Files selected for processing (6)
  • CONTRIBUTING.md
  • pyproject.toml
  • tests/evals/__init__.py
  • tests/evals/_harness.py
  • tests/evals/test_grounding.py
  • tests/evals/test_harness.py

Comment on lines +147 to +162
trace = _run(execution_agent, prompt)
assert_consulted_docs(trace)

@pytest.mark.parametrize(
"prompt",
[
"What kpoints_distance should I use for PwBandsWorkChain?",
"What ecutwfc is recommended for silicon with SSSP efficiency?",
],
)
def test_no_invented_physics_values(
self, execution_agent: Agent, prompt: str
) -> None:
"""Every cutoff or spacing stated must appear in something a tool returned."""
trace = _run(execution_agent, prompt)
assert_grounded_quantities(trace, prompt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Require citations for execution-agent documentation answers.

These cases permit a docs-derived execution answer without a user-facing citation. Require citations for both paths; the numeric case should also require documentation consultation.

Proposed fix
         trace = _run(execution_agent, prompt)
         assert_consulted_docs(trace)
+        assert_cited(trace)
 
@@
         trace = _run(execution_agent, prompt)
+        assert_consulted_docs(trace)
+        assert_cited(trace)
         assert_grounded_quantities(trace, prompt)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
trace = _run(execution_agent, prompt)
assert_consulted_docs(trace)
@pytest.mark.parametrize(
"prompt",
[
"What kpoints_distance should I use for PwBandsWorkChain?",
"What ecutwfc is recommended for silicon with SSSP efficiency?",
],
)
def test_no_invented_physics_values(
self, execution_agent: Agent, prompt: str
) -> None:
"""Every cutoff or spacing stated must appear in something a tool returned."""
trace = _run(execution_agent, prompt)
assert_grounded_quantities(trace, prompt)
trace = _run(execution_agent, prompt)
assert_consulted_docs(trace)
assert_cited(trace)
`@pytest.mark.parametrize`(
"prompt",
[
"What kpoints_distance should I use for PwBandsWorkChain?",
"What ecutwfc is recommended for silicon with SSSP efficiency?",
],
)
def test_no_invented_physics_values(
self, execution_agent: Agent, prompt: str
) -> None:
"""Every cutoff or spacing stated must appear in something a tool returned."""
trace = _run(execution_agent, prompt)
assert_consulted_docs(trace)
assert_cited(trace)
assert_grounded_quantities(trace, prompt)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/evals/test_grounding.py` around lines 147 - 162, Update the
execution-agent documentation-answer tests around
test_no_invented_physics_values and the preceding parameterized case so both
answer paths require a user-facing citation. Ensure the numeric cutoff/spacing
case continues to validate grounded quantities and additionally asserts that
documentation was consulted, while the other documentation-derived case also
asserts citation presence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant