Repository navigation
✨ make grounding structural, and stop the tools asserting what they cannot support - #48
Conversation
Testing showed the prompt approach has stopped paying. An explicit instruction
added last change -- "aggregate statistics are an Execution tool, say so and
point the user there" -- was ignored in five analysis runs out of five. In the
worst run the agent made no query at all and invented both a log line
("ecutwfc=0.05 Ry") and the domain mechanics to justify it.
What has held all along is structural: requires_approval is never bypassed,
the router's strict parser refuses garbage, the fallback lands read-only. None
of those need the model to cooperate. So do that here instead.
1. query_run_context moves to the shared tools/ top level and is registered on
the Analysis agent too. It is read-only -- plain QueryBuilder queries -- and
sat on Execution only because Execution needed it first, not for any safety
reason. "What did my past runs use" is an exploration question by nature. With
both agents holding it, a misroute stops being a failure, which is already the
stated principle for search_aiida_docs and get_process_status. The situation is
removed rather than instructed against.
The prompt rule that was ignored is replaced by documentation of the tool,
including that reconstructing these statistics from query_nodes looks workable
and is not: the cutoff is not an extras field, so the query returns nulls, and
an answer assembled from nulls is a guess wearing the shape of a finding.
2. A grounding check now runs on every reply. aiida_agents.grounding pulls each
physical quantity out of the answer and reports any that appear in no tool
output, and the CLI prints a warning naming them. It catches fabrication
whether or not the model complied.
The detector lifts the eval harness's logic into the package and the harness
now imports it, so the check guarding a shipped answer and the check guarding
the test suite cannot drift -- two expressions of one idea drifting apart is
itself a bug this project has already had.
Writing its tests found the detector would have missed the very fabrication it
was built for. Requiring the number within ~20 characters of the parameter name
misses "ecutwfc varies widely (e.g., 42 for gold, 8 for the alkali metals)":
prose does not keep a value next to its name. It is now scoped to the sentence.
Unit spellings containing a digit ("1/A") had to stop being read as quantities
of their own -- a false positive is how a warning gets ignored.
Warns rather than blocks, and stays narrow: only numbers with a unit or in a
sentence naming a simulation parameter. "Found 12 structures" is prose.
📝 WalkthroughWalkthroughThe PR adds AiiDA run-context analytics to the Analysis agent and centralizes physical-quantity grounding checks. CLI commands and the REPL now warn about unsupported numerical claims, while tests cover query integration, grounding behavior, and tool evidence extraction. ChangesRun-context analytics and grounding
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant AnalysisAgent
participant query_run_context
participant AiiDAORM
AnalysisAgent->>query_run_context: submit query_type and filters
query_run_context->>AiiDAORM: query AiiDA run-context records
AiiDAORM-->>query_run_context: return ORM results
query_run_context-->>AnalysisAgent: return structured statistics
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
There was a problem hiding this comment.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/agents/analysis/prompt.md`:
- Around line 72-73: Update the cross-reference in the introductory
aggregate-statistics guidance immediately before the “AGGREGATE STATISTICS OVER
PAST RUNS” section to point to rule 7 instead of rule 8; leave the surrounding
rules unchanged.
In `@src/aiida_agents/cli/output.py`:
- Around line 160-164: Update _warn_ungrounded so its diagnostic warning is
emitted to stderr rather than the shared Rich console stdout, preserving --raw
output as answer-only. Add or update the CLI test for _print_reply(raw=True) to
assert stdout contains only the <answer> content and excludes the
ungrounded-values warning.
In `@src/aiida_agents/grounding.py`:
- Line 46: Update the quantity extraction regex represented by _NUMBER to
tokenize optional leading plus or minus signs, ensuring values such as -5 eV
retain their sign and remain distinguishable from +5. Add a regression covering
opposite-sign evidence so warning suppression only occurs for matching signed
quantities.
- Line 121: Update the grounding calculation around quantities_in and
_numbers_in so evidence matching preserves parameter/unit context instead of
treating every numeric token as valid evidence. Exclude identifiers such as
process PKs, counts, and exit statuses from grounding, while retaining
tool-returned values that are explicitly associated with the relevant parameter
and unit; keep prompt-number subtraction behavior intact.
- Line 59: Update the _PARAM_SENTENCE regular expression in grounding.py to use
word boundaries around parameter names, preventing matches inside identifiers
such as ecutwfc_60. Add a regression test covering this identifier case and
verify that the numeric suffix is not reported as a physical claim.
In `@src/aiida_agents/tools/run_context.py`:
- Around line 270-271: Update the structure_type handling around workflow filter
construction and the response assignments to avoid defaulting or echoing
"metallic" when the caller omitted structure_type. Preserve the explicit
caller-provided structure_type filter, while leaving the existing
structure_type_filter_note behavior intact across the corresponding paths.
- Around line 364-391: Update the common_failure_modes construction in the
result-building flow to deduplicate failure_modes while preserving a
deterministic order, such as first occurrence order, before limiting the result
to three entries. Keep the existing failure_modes collection and output shape
unchanged.
- Around line 184-224: Remove the stale Analysis Agent framing from
query_run_context: revise the query_type Field description and function
docstring to describe querying run-context data directly via database-backed
context, without implying delegation or loose coupling between agents. Update
the related parameter and return documentation consistently while preserving the
existing query behavior.
- Around line 410-460: Update the available-codes result handling around the
codes_list guards: when a code_filter is provided and no rows match it, return
the filter-specific “no codes found matching” response regardless of the filter
text, while retaining the setup guidance only for unfiltered empty results. Also
remove the unreachable results-based conditional in the successful return and
use the appropriate always-successful note.
- Around line 491-511: Apply the structure_pk filter before limiting attempts to
10: iterate through the full records collection, retain only rows accepted by
the existing node structure check, and stop once 10 matching attempts have been
appended. Preserve the current attempt mapping and exception handling in the
surrounding attempts-building logic.
- Around line 470-490: Update the failed-attempts query in the QueryBuilder
filter construction to use AiiDA’s negated equality operator for
attributes.exit_status, replacing the current "!=" filter with the valid "!=="
form while preserving the existing nonzero-status behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d5fd8764-5d26-4120-85bd-244e8c3f415c
📒 Files selected for processing (15)
src/aiida_agents/agents/analysis/__init__.pysrc/aiida_agents/agents/analysis/prompt.mdsrc/aiida_agents/agents/execution/__init__.pysrc/aiida_agents/cli/commands.pysrc/aiida_agents/cli/output.pysrc/aiida_agents/cli/repl.pysrc/aiida_agents/grounding.pysrc/aiida_agents/tools/execution/__init__.pysrc/aiida_agents/tools/run_context.pytests/agents/analysis/test_analysis.pytests/agents/execution/test_integration.pytests/cli/test_commands.pytests/evals/_harness.pytests/test_grounding.pytests/tools/test_run_context.py
| - Aggregate statistics over past runs are yours too: see rule 8. | ||
| 7. AGGREGATE STATISTICS OVER PAST RUNS: |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Wrong rule number in cross-reference.
Line 72 says "see rule 8" but the aggregate-statistics guidance is rule 7 (the section immediately following, at line 73); rule 8 is "GROUNDING IN TOOL OUTPUT." This misdirects whatever reads this reference literally.
✏️ Suggested fix
- - Aggregate statistics over past runs are yours too: see rule 8.
+ - Aggregate statistics over past runs are yours too: see rule 7.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - Aggregate statistics over past runs are yours too: see rule 8. | |
| 7. AGGREGATE STATISTICS OVER PAST RUNS: | |
| - Aggregate statistics over past runs are yours too: see rule 7. | |
| 7. AGGREGATE STATISTICS OVER PAST RUNS: |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/agents/analysis/prompt.md` around lines 72 - 73, Update the
cross-reference in the introductory aggregate-statistics guidance immediately
before the “AGGREGATE STATISTICS OVER PAST RUNS” section to point to rule 7
instead of rule 8; leave the surrounding rules unchanged.
| console.print( | ||
| f"[yellow]⚠ Not found in any tool output: {values}. " | ||
| "Verify before using these values.[/yellow]" | ||
| ) | ||
| logger.warning("ungrounded quantities in reply: %s", values) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the relevant file and nearby code.
fd -a 'output\.py$' . | sed 's#^\./##'
echo '--- outline ---'
ast-grep outline src/aiida_agents/cli/output.py --view expanded 2>/dev/null || true
echo '--- relevant lines ---'
cat -n src/aiida_agents/cli/output.py | sed -n '1,230p'
echo '--- raw/console references ---'
rg -n "raw|console|_warn_ungrounded|ungrounded|stderr|stdout" src/aiida_agents/cli/output.py src -g '*.py' 2>/dev/null | head -200Repository: aiidateam/aiida-agents
Length of output: 24424
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- Rich Console stdout implementation ---'
python3 - <<'PY'
try:
from rich.console import Console
import io
except Exception as exc:
print(type(exc).__name__, exc)
raise SystemExit(0)
buf = io.StringIO()
console = Console(file=buf)
console.print("x")
print("file:", console.file)
print("stdout:", console.file == console.stdout)
print("buf after first:", buf.getvalue(), repr(buf.getvalue()))
console.print("y", end="\n\n")
print("stdout:", console.file == console.stdout)
print("buf after second:", buf.getvalue())
PY
echo '--- CLI command relevant lines ---'
cat -n src/aiida_agents/cli/commands.py | sed -n '90,155p'
echo '--- tests for raw/uncorrupted stdout ---'
rg -n "raw|stdout|answer\.md|pipe|warn|ungrounded" tests -g '*.py' | head -200Repository: aiidateam/aiida-agents
Length of output: 10351
Keep --raw stdout free of diagnostics.
_print_reply(raw=raw) writes only <answer> to stdout, but _warn_ungrounded writes ⚠ ... via the shared Rich console after it, so aiida-agents ask ... --raw can append warning text to the captured Markdown. Emit the warning on stderr and add a --raw stdout assertion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/cli/output.py` around lines 160 - 164, Update
_warn_ungrounded so its diagnostic warning is emitted to stderr rather than the
shared Rich console stdout, preserving --raw output as answer-only. Add or
update the CLI test for _print_reply(raw=True) to assert stdout contains only
the <answer> content and excludes the ungrounded-values warning.
| r"press_conv_thr|nbnd|electron_maxstep|cutoff" | ||
| ) | ||
|
|
||
| _NUMBER = r"\d+(?:\.\d+)?(?:[eE][-+]?\d+)?" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve the quantity sign.
-5 eV is extracted as 5.0, so evidence containing +5 incorrectly suppresses the warning. Tokenize signed literals and add a regression for opposite-sign evidence.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/grounding.py` at line 46, Update the quantity extraction
regex represented by _NUMBER to tokenize optional leading plus or minus signs,
ensuring values such as -5 eV retain their sign and remain distinguishable from
+5. Add a regression covering opposite-sign evidence so warning suppression only
occurs for matching signed quantities.
| #: metals)" puts 22 characters between them, and the 8 much further. Prose does | ||
| #: not keep a value next to its name. Scoping to the sentence matches how the | ||
| #: claim is actually written. | ||
| _PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Bound parameter names to avoid identifier false positives.
ecutwfc_60 matches ecutwfc, then the sentence scan reports 60 as a physical claim. This contradicts the stated identifier exclusion. Add word boundaries and a regression case.
Proposed fix
-_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)
+_PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| _PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE) | |
| _PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/grounding.py` at line 59, Update the _PARAM_SENTENCE regular
expression in grounding.py to use word boundaries around parameter names,
preventing matches inside identifiers such as ecutwfc_60. Add a regression test
covering this identifier case and verify that the numeric suffix is not reported
as a physical claim.
| Normalised numeric literals with no source. Empty means every physics | ||
| number in the answer is traceable to a tool or to the question. | ||
| """ | ||
| return quantities_in(answer) - _numbers_in(evidence) - _numbers_in(prompt) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Match evidence claims, not arbitrary numeric tokens.
A tool return such as Process pk 60 finished makes Use ecutwfc 60 Ry appear grounded. PKs, counts, and exit statuses can therefore bless fabricated parameters. Preserve parameter/unit context when extracting evidence, rather than subtracting every number in the tool text.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/grounding.py` at line 121, Update the grounding calculation
around quantities_in and _numbers_in so evidence matching preserves
parameter/unit context instead of treating every numeric token as valid
evidence. Exclude identifiers such as process PKs, counts, and exit statuses
from grounding, while retaining tool-returned values that are explicitly
associated with the relevant parameter and unit; keep prompt-number subtraction
behavior intact.
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/agents/analysis/prompt.md`:
- Around line 72-73: Update the cross-reference in the introductory
aggregate-statistics guidance immediately before the “AGGREGATE STATISTICS OVER
PAST RUNS” section to point to rule 7 instead of rule 8; leave the surrounding
rules unchanged.
In `@src/aiida_agents/cli/output.py`:
- Around line 160-164: Update _warn_ungrounded so its diagnostic warning is
emitted to stderr rather than the shared Rich console stdout, preserving --raw
output as answer-only. Add or update the CLI test for _print_reply(raw=True) to
assert stdout contains only the <answer> content and excludes the
ungrounded-values warning.
In `@src/aiida_agents/grounding.py`:
- Line 46: Update the quantity extraction regex represented by _NUMBER to
tokenize optional leading plus or minus signs, ensuring values such as -5 eV
retain their sign and remain distinguishable from +5. Add a regression covering
opposite-sign evidence so warning suppression only occurs for matching signed
quantities.
- Line 121: Update the grounding calculation around quantities_in and
_numbers_in so evidence matching preserves parameter/unit context instead of
treating every numeric token as valid evidence. Exclude identifiers such as
process PKs, counts, and exit statuses from grounding, while retaining
tool-returned values that are explicitly associated with the relevant parameter
and unit; keep prompt-number subtraction behavior intact.
- Line 59: Update the _PARAM_SENTENCE regular expression in grounding.py to use
word boundaries around parameter names, preventing matches inside identifiers
such as ecutwfc_60. Add a regression test covering this identifier case and
verify that the numeric suffix is not reported as a physical claim.
In `@src/aiida_agents/tools/run_context.py`:
- Around line 270-271: Update the structure_type handling around workflow filter
construction and the response assignments to avoid defaulting or echoing
"metallic" when the caller omitted structure_type. Preserve the explicit
caller-provided structure_type filter, while leaving the existing
structure_type_filter_note behavior intact across the corresponding paths.
- Around line 364-391: Update the common_failure_modes construction in the
result-building flow to deduplicate failure_modes while preserving a
deterministic order, such as first occurrence order, before limiting the result
to three entries. Keep the existing failure_modes collection and output shape
unchanged.
- Around line 184-224: Remove the stale Analysis Agent framing from
query_run_context: revise the query_type Field description and function
docstring to describe querying run-context data directly via database-backed
context, without implying delegation or loose coupling between agents. Update
the related parameter and return documentation consistently while preserving the
existing query behavior.
- Around line 410-460: Update the available-codes result handling around the
codes_list guards: when a code_filter is provided and no rows match it, return
the filter-specific “no codes found matching” response regardless of the filter
text, while retaining the setup guidance only for unfiltered empty results. Also
remove the unreachable results-based conditional in the successful return and
use the appropriate always-successful note.
- Around line 491-511: Apply the structure_pk filter before limiting attempts to
10: iterate through the full records collection, retain only rows accepted by
the existing node structure check, and stop once 10 matching attempts have been
appended. Preserve the current attempt mapping and exception handling in the
surrounding attempts-building logic.
- Around line 470-490: Update the failed-attempts query in the QueryBuilder
filter construction to use AiiDA’s negated equality operator for
attributes.exit_status, replacing the current "!=" filter with the valid "!=="
form while preserving the existing nonzero-status behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d5fd8764-5d26-4120-85bd-244e8c3f415c
📒 Files selected for processing (15)
src/aiida_agents/agents/analysis/__init__.pysrc/aiida_agents/agents/analysis/prompt.mdsrc/aiida_agents/agents/execution/__init__.pysrc/aiida_agents/cli/commands.pysrc/aiida_agents/cli/output.pysrc/aiida_agents/cli/repl.pysrc/aiida_agents/grounding.pysrc/aiida_agents/tools/execution/__init__.pysrc/aiida_agents/tools/run_context.pytests/agents/analysis/test_analysis.pytests/agents/execution/test_integration.pytests/cli/test_commands.pytests/evals/_harness.pytests/test_grounding.pytests/tools/test_run_context.py
🛑 Comments failed to post (6)
src/aiida_agents/tools/run_context.py (6)
184-224: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Stale "asks the Analysis Agent" framing in the tool's own docs.
The module docstring (lines 9-15) explicitly says this tool was renamed away from
query_analysis_agentbecause a docstring claiming it "asked the Analysis agent" was never true — it runsQueryBuilderqueries directly. But theFielddescription ("What to ask Analysis Agent...") and this function's own docstring ("Query the Analysis Agent for context...loose coupling between Execution Agent and Analysis Agent...instead of Execution Agent importing Analysis Agent's tools directly") still carry that exact framing. This is now also registered directly on the Analysis agent's own toolset, so the model reading "what to ask Analysis Agent" while being the Analysis agent itself is genuinely confusing — the same failure mode the rename was meant to fix.✏️ Suggested doc fix
query_type: t.Annotated[ str, Field( description=( - "What to ask Analysis Agent. Options: " + "What run-context statistic to retrieve from the active AiiDA " + "profile. Options: " "'past_successful_workflows', 'available_codes', " "'failed_attempts', 'available_pseudos'" ) ), ],- """Query the Analysis Agent for context before generating a workflow spec. - - This tool provides loose coupling between Execution Agent and Analysis Agent. - Instead of Execution Agent importing Analysis Agent's tools directly, it calls - query_run_context() and gets back structured context. - - The Execution Agent uses this context to: + """Query the active AiiDA profile for run-context statistics. + + Both the Analysis and Execution agents call this directly (it runs + ``QueryBuilder`` queries in-process; see the module docstring). Callers + use this context to:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def query_run_context( query_type: t.Annotated[ str, Field( description=( "What run-context statistic to retrieve from the active AiiDA " "profile. Options: " "'past_successful_workflows', 'available_codes', " "'failed_attempts', 'available_pseudos'" ) ), ], filters: t.Annotated[ dict[str, t.Any], Field( description=( "Context-specific filters (e.g. structure_type, composition, workflow_type). " "Note: 'structure_type' is not a database-level filter — AiiDA stores no " "per-node material-class attribute. It is echoed back as metadata only." ) ), ], ) -> dict[str, t.Any]: """Query the active AiiDA profile for run-context statistics. Both the Analysis and Execution agents call this directly (it runs ``QueryBuilder`` queries in-process; see the module docstring). Callers use this context to: 1. Learn what parameters worked for similar structures in the past. 2. Check what codes are available. 3. Understand why similar setups failed. 4. Make better parameter choices. Args: query_type: The kind of information to retrieve. filters: Context that narrows the query (workflow_type, structure_type, etc.). Returns: dict with context-specific data (past runs, codes, failures, etc.). """🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 184 - 224, Remove the stale Analysis Agent framing from query_run_context: revise the query_type Field description and function docstring to describe querying run-context data directly via database-backed context, without implying delegation or loose coupling between agents. Update the related parameter and return documentation consistently while preserving the existing query behavior.
270-271: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Unrequested
structure_typedefault is echoed back as if it were the real filter.
structure_typedefaults to"metallic"when the caller omits it, then that default is echoed verbatim intores["structure_type"]even though the accompanyingstructure_type_filter_notealready says it isn't a real filter. A caller that didn't ask about metals will still see"structure_type": "metallic"in the response, which is exactly the kind of unsourced-looking value this PR's grounding effort is trying to eliminate.✏️ Suggested fix
- structure_type = filters.get("structure_type", "metallic") + structure_type = filters.get("structure_type")Also applies to: 293-294, 378-379
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 270 - 271, Update the structure_type handling around workflow filter construction and the response assignments to avoid defaulting or echoing "metallic" when the caller omitted structure_type. Preserve the explicit caller-provided structure_type filter, while leaving the existing structure_type_filter_note behavior intact across the corresponding paths.
364-391: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
common_failure_modesordering is nondeterministic across process runs.
list(set(failure_modes))[:3]dedupes via aset, whose string iteration order depends on Python's per-process hash seed (randomized by default). Two identical databases queried from two separate process launches can return differentcommon_failure_modes, undermining reproducibility of a tool whose whole point is to be a trustworthy source of truth.✏️ Suggested fix
- "common_failure_modes": list(set(failure_modes))[:3], + "common_failure_modes": list(dict.fromkeys(failure_modes))[:3],📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.failure_modes: list[str] = [] for row in failed_runs[:10]: try: fnode = orm.load_node(row[1]) if fnode.exit_message: failure_modes.append(f"Exit status {row[0]}: {fnode.exit_message}") else: failure_modes.append(f"Exit status {row[0]}") except Exception as exc: logger.debug("Could not load failed node %s: %s", row[1], exc) return { "query_type": "past_successful_workflows", "workflow_type": workflow_type, "structure_type": structure_type, "structure_type_filter_note": ( "structure_type is not filterable at the database level; " "all finished workflows of this type are included." ), "count": len(successful_runs), "success_rate": success_rate, "median_ecutwfc": med_ecutwfc, "median_kpoints_distance": med_kpoints, "units": _UNITS, "common_parameters": {"ecutwfc": med_ecutwfc} if med_ecutwfc else {}, "common_failure_modes": list(dict.fromkeys(failure_modes))[:3], "example_structures": example_structs, "note": f"Analyzed {total_runs} historical workflow(s) in active database ({len(successful_runs)} successful).",🧰 Tools
🪛 Ruff (0.16.0)
[warning] 372-372: Do not catch blind exception:
Exception(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 364 - 391, Update the common_failure_modes construction in the result-building flow to deduplicate failure_modes while preserving a deterministic order, such as first occurrence order, before limiting the result to three entries. Keep the existing failure_modes collection and output shape unchanged.
410-460: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
"No codes matching filter" message only fires when the filter string literally contains "unknown".
The branch at line 432 that reports
f"No codes found matching {code_filter!r}"is gated oncode_filter and "unknown" in str(code_filter).lower()— a heuristic that only matches the test fixture's literal"unknown-abinit-code"string. A real filter like"abinit-v2"that matches nothing falls through to the generic440-449branch and tells the caller "No codes found in the active AiiDA profile. Please set up a code first," which is false when codes exist but just didn't match the filter — a misleading answer that could send the agent down the wrong path (e.g. telling the user to install a code that's already installed under a different label). Separately, theif results else ...at line 457 is unreachable dead code: by that pointcodes_list(and henceresults, its superset) is already known non-empty from the two guards above.🐛 Suggested fix
- if not codes_list and code_filter and "unknown" in str(code_filter).lower(): - return { - "query_type": "available_codes", - "codes": [], - "recommended_version": None, - "note": f"No codes found matching {code_filter!r} in active AiiDA profile.", - } - - if not codes_list: + if not codes_list: + if code_filter: + note = f"No codes found matching {code_filter!r} in active AiiDA profile." + else: + note = ( + "No codes found in the active AiiDA profile. " + "Please set up a code first: verdi code setup" + ) return { "query_type": "available_codes", "codes": [], "recommended_version": None, - "note": ( - "No codes found in the active AiiDA profile. " - "Please set up a code first: verdi code setup" - ), + "note": note, } return { "query_type": "available_codes", "codes": codes_list, "recommended_version": recommended, - "note": ( - f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile." - if results - else "No matching codes found in active AiiDA profile. Providing schema recommendation." - ), + "note": f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile.", }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.code_filter = ( filters.get("code") or filters.get("plugin") or filters.get("code_label") ) codes_list: list[dict[str, str]] = [] recommended: str | None = None for row in results: label, plugin, desc = row[0], row[1], row[2] if code_filter and ( str(code_filter).lower() not in str(label).lower() and str(code_filter).lower() not in str(plugin).lower() ): continue codes_list.append( { "label": str(label), "plugin": str(plugin) if plugin else "unknown", "description": str(desc) if desc else "", } ) if not recommended or "pw" in str(label).lower(): recommended = str(label) if not codes_list: if code_filter: note = f"No codes found matching {code_filter!r} in active AiiDA profile." else: note = ( "No codes found in the active AiiDA profile. " "Please set up a code first: verdi code setup" ) return { "query_type": "available_codes", "codes": [], "recommended_version": None, "note": note, } return { "query_type": "available_codes", "codes": codes_list, "recommended_version": recommended, "note": f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile.", }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 410 - 460, Update the available-codes result handling around the codes_list guards: when a code_filter is provided and no rows match it, return the filter-specific “no codes found matching” response regardless of the filter text, while retaining the setup guidance only for unfiltered empty results. Also remove the unreachable results-based conditional in the successful return and use the appropriate always-successful note.
470-490: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does AiiDA's orm.QueryBuilder support "!=" as a filter operator, or only "!=="?💡 Result:
AiiDA's QueryBuilder supports the inequality operator by prepending an exclamation mark (!) to the equality operator (==), resulting in!== [1][2][3]. While you can use the!== syntax to explicitly represent "not equal to," the QueryBuilder's filter system is designed so that any supported operator can be negated by adding a! prefix [1][3]. In practice,!== is the standard way to express "not equal" for the equality operator, which the QueryBuilder subsequently translates into a standard!= SQL operator [1][3]. Additionally, as of AiiDA 2.6, a new programmatic filter negation syntax was introduced allowing for the use of the ~ operator (e.g., ~(field == value)), which provides an alternative way to express negation [2].
Citations:
- 1: https://aiida.readthedocs.io/projects/aiida-core/en/stable/topics/database.html
- 2: https://aiida.readthedocs.io/projects/aiida-core/en/stable/howto/query.html
- 3: https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/database.html
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file excerpt ==" wc -l src/aiida_agents/tools/run_context.py sed -n '440,510p' src/aiida_agents/tools/run_context.py echo echo "== dependency/version hints ==" git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|Pipfile|poetry\.lock|uv\.lock|setup\.py|setup\.cfg)$' || true for f in $(git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|Pipfile|poetry\.lock|uv\.lock|setup\.py|setup\.cfg)$' || true); do echo "--- $f ---" rg -n "aiida|aiida-core|alembic|pydantic|pytest" "$f" || true done echo echo "== prompt guidance occurrence ==" rg -n 'Not-equal is|!=|!==|QueryBuilder|failed attempts' -S .Repository: aiidateam/aiida-agents
Length of output: 50378
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== tests mentioning failed_attempts / query_failed_attempts ==" rg -n "failed_attempts|query_failed_attempts|failed attempts|exit_status|exit_status|attempts" tests src -g 'test*.py' -g '*.py' || true echo echo "== relevant prompt / guidance ==" rg -n -C 3 'Not-equal is|!=|!==|QueryBuilder|failed attempts' prompt.md CONTRIBUTING.md src/aiida_agents -g '!*.pyc' || true echo echo "== locate QueryBuilder operator validation within source / lock ==" tmpdir="$(mktemp -d)" cd "$tmpdir" git clone --depth 1 --branch v3.9.0 https://github.com/aiidateam/aiida-core.git >/tmp/clone.log 2>&1 || true if [ -d aiida-core ]; then cd aiida-core rg -n "!==|!=".*"?=|OPERATOR|operators|validate_operator|QueryBuilder" aiida -g '*.py' | head -n 200 || true else echo "aiida-core checkout unavailable" fiRepository: aiidateam/aiida-agents
Length of output: 152
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== local test and prompt references (fast, targeted) ==" python3 /bin/bash - <<'PY' import subprocess result = subprocess.run( ["rg", "-n", "-C", "2", "query_failed_attempts|failed_attempts|Not-equal is|Not equal|Not equal|!=.*:|!==|QueryBuilderDict|QueryBuilder"], "--glob", "**/*.py", "--glob", "**/*.md", pytest, capture_output=True, text=True, timeout=60 ) PY echo echo "== inspect query_failed_attempts call sites only ==" rg -n "query_failed_attempts" src tests -g '*.py' || trueRepository: aiidateam/aiida-agents
Length of output: 374
🌐 Web query:
AiiDA QueryBuilder negation operator '!=' unsupported '=='💡 Result:
In AiiDA QueryBuilder, the
!=operator is not directly supported as a standalone operator. Instead, to perform a negation (such as "not equal to"), you must prepend an exclamation mark!to the corresponding operator [1][2]. To express "not equal to," you use the negation of the equality operator==, which is written as!==[1][2][3]. Example usage: filters={'some_field': {'!==': 'value_to_exclude'}} Additionally, for newer versions of AiiDA (2.6 and later), a logical expression syntax allows for the use of the~operator to negate any expression, such as~ (Node.fields.some_field == 'value')[1][2][3]. You can also use~==in some contexts to represent inequality [4].Citations:
- 1: https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/database.html
- 2: https://aiida.readthedocs.io/projects/aiida-core/en/stable/howto/query.html
- 3: https://aiida.readthedocs.io/projects/aiida-core/en/stable/topics/database.html
- 4: https://aiida.readthedocs.io/projects/aiida-core/en/stable/reference/apidoc/aiida.storage.psql_dos.orm.querybuilder.html
Use AiiDA’s inequality operator for
failed_attempts.
{"!=": 0}is not a validQueryBuilderfilter operator; write the negated equality operator as{"!==": 0}. Otherwise the query raises, the surroundingexcept Exceptionswallows it, and the tool reports no failed attempts even when failed attempts exist.🧰 Tools
🪛 Ruff (0.16.0)
[warning] 487-487: Do not catch blind exception:
Exception(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 470 - 490, Update the failed-attempts query in the QueryBuilder filter construction to use AiiDA’s negated equality operator for attributes.exit_status, replacing the current "!=" filter with the valid "!==" form while preserving the existing nonzero-status behavior.
491-511: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
structure_pkpost-filter is applied after truncating to the first 10 raw records.
for row in records[:10]:slices the unfiltered query results to 10 rows before thestructure_pkmatch check runs. If a database has more than 10 failed attempts and the ones matchingstructure_pkare not among the first 10 (arbitrary DB order, noorder_by), they are never inspected — the caller can get zero or fewer than 10 matching attempts even though more exist. This defeats the purpose of thestructure_pkfilter for any database with a non-trivial number of failures.🐛 Suggested fix
- attempts: list[dict[str, t.Any]] = [] - for row in records[:10]: + attempts: list[dict[str, t.Any]] = [] + for row in records: + if len(attempts) >= 10: + break pk, status, msg = row[0], row[1], row[2]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.attempts: list[dict[str, t.Any]] = [] for row in records: if len(attempts) >= 10: break pk, status, msg = row[0], row[1], row[2] if structure_pk: try: node = orm.load_node(pk) if "structure" in node.inputs and str(node.inputs.structure.pk) != str( structure_pk ): continue except Exception as exc: logger.debug("Could not load node %s for structure filter: %s", pk, exc) continue attempts.append( { "pk": pk, "exit_code": status, "failure_reason": msg or f"Failed with exit status {status}", } )🧰 Tools
🪛 Ruff (0.16.0)
[warning] 501-501: Do not catch blind exception:
Exception(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/run_context.py` around lines 491 - 511, Apply the structure_pk filter before limiting attempts to 10: iterate through the full records collection, retain only rows accepted by the existing node structure check, and stop once 10 matching attempts have been appended. Preserve the current attempt mapping and exception handling in the surrounding attempts-building logic.
Two commits. The first replaces a prompt rule that measurably did not work; the second is an audit for fabricated data that found four more, all in the tool rather than the model.
1. Grounding becomes structural (
59aae7a)#47 added an explicit instruction — "aggregate statistics are an Execution tool, say so and point the user there". It was ignored in five analysis runs out of five. In the worst run the agent made no query at all and invented both a log line (
ecutwfc=0.05 Ry) and the domain mechanics to justify it.What has held is structural:
requires_approvalnever bypassed, the router's strict parser refusing garbage, the fallback landing read-only. None need the model to cooperate.Both agents now hold
query_run_context. It is read-only — plainQueryBuilderqueries — and sat on Execution only because Execution needed it first. "What did my past runs use" is an exploration question by nature. With both agents holding it, a misroute stops being a failure, which is already the principle behind both sharingsearch_aiida_docs. The situation is removed rather than instructed against.A grounding check runs on every reply.
aiida_agents.groundingreports any physical quantity in the answer that appears in no tool output, and the CLI prints:The detector lifts the eval harness's logic into the package and the harness now imports it, so the check guarding a shipped answer and the one guarding the test suite cannot drift — two expressions of one idea drifting apart is a bug this project has already had (#37).
Writing its tests found the detector would have missed the fabrication it was built for: requiring the number within ~20 characters of the parameter name misses
ecutwfc varies widely (e.g., 42 for gold, 8 for the alkali metals). Prose does not keep a value next to its name. Now scoped to the sentence.2. The tools were fabricating too (
210e4ad)An audit for fake data found four values
query_run_contextsupplied itself, which a caller could not tell apart from queried ones. The agents' grounding rules forbid exactly this of the model — it was enforced everywhere except in the tool feeding it.structure_type"metallic"and returned as though asked for. Filters nothing; the response even carried a note saying so. A model could report "for metallic structures, the median is…" on a qualifier from neither the user nor the database.workflow_typeaiida.workflows:PwRelaxWorkChain. Omitting it gave confident statistics about a workflow nobody named — on a profile with 35 entry points, a wrong answer delivered well.recommended_versionrecommended_family"SSSP/1.3/PBE/efficiency (needs installation)"— a family label for something not in the profile, in a field a model would reasonably pass on as a workflow input.All four removed.
workflow_typeis now required, raising aValueErrorthat names the fix, whichRetryOnToolErrorturns into a correction the model can act on. The pseudos note still gives theaiida-pseudo installcommand — guidance in prose, not a usable-looking reference.Test plan
ruff/ruff format/mypyclean across 113 files60matching60.0, and unit spellings containing a digit (1/A)all_messages()rather than the check being made tolerant of its absence — a check that skips silently when an interface differs is the fail-open pattern this PR removesWhat this does not claim
The warning does not stop a model fabricating; it makes fabrication visible when it reaches the user. And the opt-in eval tier's 19 cases have still never produced a complete run, so CONTRIBUTING's claim that LLM boundaries are "validated by the eval harness" remains half true — the CI tier is real, the model-facing tier is unproven.