Repository navigation
🧪 evals: assert on what an agent did, not on what it said - #40
Conversation
Every grounding defect this project has hit was found by a human running the CLI and noticing the answer contained a number nobody had retrieved: the false-healthy RAG status, the weak-hit answering, the execution agent routing past the docs. None of them was catchable by the existing suite, which exercises tools in isolation while the defects live in the agent's behaviour -- which tool it reached for, and whether its prose is traceable to what that tool returned. CONTRIBUTING already claimed these boundaries were "validated by the eval harness"; there was no eval harness. Add tests/evals. A run is recorded into a RunTrace (tool calls in order, each one's output, the final answer) via capture_run_messages, and asserted on: - assert_consulted_docs -- did it look anything up before answering, the routing defect expressed as an assertion. - assert_grounded_quantities -- does every physical quantity in the answer appear in some tool output. Only numbers carrying a unit or bound to a named simulation parameter count, so "step 2 of 3" is not mistaken for a physics claim, and 60 in the answer matches 60.0 in a tool return rather than being reported as invented. - assert_cited -- an answer built on retrieved docs that carries no citation is a model that retrieved and then answered from memory anyway. Two tiers. test_harness.py runs in CI on FunctionModel: it scripts a fabricating agent and requires each check to catch it, then scripts the correct behaviour and requires each to pass, so the assertions cannot rot into always-passing. test_grounding.py is opt-in (AIIDA_AGENTS_EVAL=1, marked llm), needs a real model and a built index, and covers both directions of the routing fix -- knowledge questions must reach the docs, and setup requests must still discover entry points rather than detouring into them.
📝 WalkthroughWalkthroughAdds a grounding evaluation harness that records agent tool traces, checks evidence and citations, validates behavior with deterministic tests, and runs opt-in real-model evaluations. Documentation and pytest configuration describe the command, marker, environment, and evaluation tiers. ChangesGrounding evaluations
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant EvalTest
participant Agent
participant SearchTool
participant RunTrace
participant Assertions
EvalTest->>Agent: run prompt
Agent->>SearchTool: search documentation
SearchTool-->>Agent: return evidence
Agent-->>RunTrace: record tool calls and answer
RunTrace->>Assertions: validate quantities, retrieval, and citations
Assertions-->>EvalTest: pass or fail evaluation
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The first real-model run of the opt-in tier failed all ten cases without producing any grounding signal: every request went to a local Ollama endpoint instead of the configured provider. tests/conftest.py chdirs every test into an empty temp directory -- deliberately, so a developer's .env cannot leak into hermetic tests -- and pydantic-settings resolves .env relative to the working directory. The eval fixtures inherited that, found no .env, and fell back to ModelSettings' bare defaults (ollama/qwen3.5:2b), which then exhausted its context without answering. Ten failures for a model that was never contacted. Copy the project's .env into the test's cwd, which is what the conftest docstring already prescribes for a test that needs one. The fixture depends on _isolate_cwd by name rather than trusting fixture ordering, since the copy has to land after the chdir or it goes to the repository root. Exported shell variables survive the chdir and are unaffected; they still take precedence, as pydantic-settings intends. Also log the resolved provider/model on every case, passing or failing. The underlying problem was not the fallback but its invisibility: a grounding result is meaningless without knowing which model produced it, and reading ten failures as model weakness when the model was never reached is exactly the wrong conclusion to invite. Tested in CI rather than only in the opt-in tier: the new cases assert the bare defaults are what an empty cwd yields, then that the copy makes the configured values win -- the bug and the fix pinned in one test.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/evals/test_grounding.py`:
- Around line 147-162: Update the execution-agent documentation-answer tests
around test_no_invented_physics_values and the preceding parameterized case so
both answer paths require a user-facing citation. Ensure the numeric
cutoff/spacing case continues to validate grounded quantities and additionally
asserts that documentation was consulted, while the other documentation-derived
case also asserts citation presence.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ef1fce60-d1d0-4e1b-95fb-9cd379563e5e
📒 Files selected for processing (6)
CONTRIBUTING.mdpyproject.tomltests/evals/__init__.pytests/evals/_harness.pytests/evals/test_grounding.pytests/evals/test_harness.py
| trace = _run(execution_agent, prompt) | ||
| assert_consulted_docs(trace) | ||
|
|
||
| @pytest.mark.parametrize( | ||
| "prompt", | ||
| [ | ||
| "What kpoints_distance should I use for PwBandsWorkChain?", | ||
| "What ecutwfc is recommended for silicon with SSSP efficiency?", | ||
| ], | ||
| ) | ||
| def test_no_invented_physics_values( | ||
| self, execution_agent: Agent, prompt: str | ||
| ) -> None: | ||
| """Every cutoff or spacing stated must appear in something a tool returned.""" | ||
| trace = _run(execution_agent, prompt) | ||
| assert_grounded_quantities(trace, prompt) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require citations for execution-agent documentation answers.
These cases permit a docs-derived execution answer without a user-facing citation. Require citations for both paths; the numeric case should also require documentation consultation.
Proposed fix
trace = _run(execution_agent, prompt)
assert_consulted_docs(trace)
+ assert_cited(trace)
@@
trace = _run(execution_agent, prompt)
+ assert_consulted_docs(trace)
+ assert_cited(trace)
assert_grounded_quantities(trace, prompt)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| trace = _run(execution_agent, prompt) | |
| assert_consulted_docs(trace) | |
| @pytest.mark.parametrize( | |
| "prompt", | |
| [ | |
| "What kpoints_distance should I use for PwBandsWorkChain?", | |
| "What ecutwfc is recommended for silicon with SSSP efficiency?", | |
| ], | |
| ) | |
| def test_no_invented_physics_values( | |
| self, execution_agent: Agent, prompt: str | |
| ) -> None: | |
| """Every cutoff or spacing stated must appear in something a tool returned.""" | |
| trace = _run(execution_agent, prompt) | |
| assert_grounded_quantities(trace, prompt) | |
| trace = _run(execution_agent, prompt) | |
| assert_consulted_docs(trace) | |
| assert_cited(trace) | |
| `@pytest.mark.parametrize`( | |
| "prompt", | |
| [ | |
| "What kpoints_distance should I use for PwBandsWorkChain?", | |
| "What ecutwfc is recommended for silicon with SSSP efficiency?", | |
| ], | |
| ) | |
| def test_no_invented_physics_values( | |
| self, execution_agent: Agent, prompt: str | |
| ) -> None: | |
| """Every cutoff or spacing stated must appear in something a tool returned.""" | |
| trace = _run(execution_agent, prompt) | |
| assert_consulted_docs(trace) | |
| assert_cited(trace) | |
| assert_grounded_quantities(trace, prompt) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/evals/test_grounding.py` around lines 147 - 162, Update the
execution-agent documentation-answer tests around
test_no_invented_physics_values and the preceding parameterized case so both
answer paths require a user-facing citation. Ensure the numeric cutoff/spacing
case continues to validate grounded quantities and additionally asserts that
documentation was consulted, while the other documentation-derived case also
asserts citation presence.
Every grounding defect this project has hit was found by a human running the CLI and noticing the answer contained a number nobody had retrieved — the false-healthy RAG status (#37), the weak-hit answering and the execution agent routing past the docs (#39). None of them was catchable by the existing suite, which exercises tools in isolation while the defects live in the agent's behaviour: which tool it reached for, and whether its prose is traceable to what that tool returned.
CONTRIBUTING.mdalready claimed these boundaries were "validated by the eval harness". There was no eval harness. This adds one.What it asserts
A run is recorded into a
RunTrace— tool calls in order, each call's output, and the final answer — viacapture_run_messages. Three checks, each corresponding to a failure that actually shipped:assert_consulted_docs— did it look anything up before answering? The 🐛 rag: stop the agents answering from memory when the docs don't cover it #39 routing defect, expressed as an assertion instead of something a human has to notice.assert_grounded_quantities— does every physical quantity in the answer appear in some tool output? Only numbers carrying a unit or bound to a named simulation parameter count, so"step 2 of 3"isn't mistaken for a physics claim; and values are normalised, so60in the answer matches60.0in a tool return rather than being reported as invented. Without both filters the check cries wolf and stops being believed.assert_cited— an answer built on retrieved docs carrying no citation is a model that retrieved and then answered from memory anyway.Grounding is checked against all tool output, not just the documentation search: a k-point spacing taken from
build_workflow_inputsis grounded just as legitimately as one quoted from the docs.Two tiers
tests/evals/test_harness.py— runs in CI, offline, deterministic. It tests the assertions themselves. An eval that quietly degrades into always-passing is worse than none, because it certifies nothing while looking green. So it scripts a fabricating agent withFunctionModel— searches, gets a real excerpt, then states a spacing that appears nowhere in it — and requires the check to catch that, then requires a correctly-quoted answer to pass. Same for skipped retrieval and missing citations.tests/evals/test_grounding.py— opt-in (AIIDA_AGENTS_EVAL=1, markedllm). Needs a real model and a built index, so it never runs in CI. Covers both directions of the #39 routing fix: knowledge questions must reach the docs, and setup requests must still discover entry points rather than detouring into them — splitting the two apart could easily over-correct, and nothing else would catch that.The opt-in tier is flaky by nature. A weaker model fails it more often, and that is a result rather than a broken test — the module docstring says so explicitly, so nobody later "fixes" a failure by loosening an assertion.
Test plan
ruff/ruff format/mypy/uv-lockclean viapre-commitmainAIIDA_AGENTS_EVAL=1; thellmmarker is registered inpyproject.toml, so no unknown-marker warningCONTRIBUTING.mdgains a section on when to run it: after changing a system prompt, a tool description, or anything insrc/aiida_agents/rag/.