Skip to content

✨ make grounding structural, and stop the tools asserting what they cannot support - #48

Merged
Jaweria-B merged 1 commit into
mainfrom
feat/structural-grounding
Jul 29, 2026
Merged

Jaweria-B merged 1 commit into
mainfrom
feat/structural-grounding

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator

Two commits. The first replaces a prompt rule that measurably did not work; the second is an audit for fabricated data that found four more, all in the tool rather than the model.


1. Grounding becomes structural (59aae7a)

#47 added an explicit instruction — "aggregate statistics are an Execution tool, say so and point the user there". It was ignored in five analysis runs out of five. In the worst run the agent made no query at all and invented both a log line (ecutwfc=0.05 Ry) and the domain mechanics to justify it.

What has held is structural: requires_approval never bypassed, the router's strict parser refusing garbage, the fallback landing read-only. None need the model to cooperate.

Both agents now hold query_run_context. It is read-only — plain QueryBuilder queries — and sat on Execution only because Execution needed it first. "What did my past runs use" is an exploration question by nature. With both agents holding it, a misroute stops being a failure, which is already the principle behind both sharing search_aiida_docs. The situation is removed rather than instructed against.

A grounding check runs on every reply. aiida_agents.grounding reports any physical quantity in the answer that appears in no tool output, and the CLI prints:

⚠ Not found in any tool output: 0.03, 0.05. Verify before using these values.

The detector lifts the eval harness's logic into the package and the harness now imports it, so the check guarding a shipped answer and the one guarding the test suite cannot drift — two expressions of one idea drifting apart is a bug this project has already had (#37).

Writing its tests found the detector would have missed the fabrication it was built for: requiring the number within ~20 characters of the parameter name misses ecutwfc varies widely (e.g., 42 for gold, 8 for the alkali metals). Prose does not keep a value next to its name. Now scoped to the sentence.


2. The tools were fabricating too (210e4ad)

An audit for fake data found four values query_run_context supplied itself, which a caller could not tell apart from queried ones. The agents' grounding rules forbid exactly this of the model — it was enforced everywhere except in the tool feeding it.

Value What it actually was
structure_type Defaulted to "metallic" and returned as though asked for. Filters nothing; the response even carried a note saying so. A model could report "for metallic structures, the median is…" on a qualifier from neither the user nor the database.
workflow_type Defaulted to aiida.workflows:PwRelaxWorkChain. Omitting it gave confident statistics about a workflow nobody named — on a profile with 35 entry points, a wrong answer delivered well.
recommended_version "The first code found, or any whose label contains 'pw'." Not a recommendation; the name asserted an endorsement the tool has no basis for; and it held a label, not a version. The branch that would have said "Providing schema recommendation" was also unreachable.
recommended_family With none installed, returned the literal "SSSP/1.3/PBE/efficiency (needs installation)" — a family label for something not in the profile, in a field a model would reasonably pass on as a workflow input.

All four removed. workflow_type is now required, raising a ValueError that names the fix, which RetryOnToolError turns into a correction the model can act on. The pseudos note still gives the aiida-pseudo install command — guidance in prose, not a usable-looking reference.

Test plan

  • ruff / ruff format / mypy clean across 113 files
  • Full suite: 563 passed, 30 skipped
  • The two real fabrications from testing are test cases: the number bolted onto a retrieved label, and the invented log line from a run with no query
  • Each removed value has a regression test naming what it used to invent
  • False positives pinned — prose counts, PKs, exit statuses, the user's own supplied value, 60 matching 60.0, and unit spellings containing a digit (1/A)
  • Pinned tool surfaces (agent + MCP) updated deliberately
  • The CLI test fake gained all_messages() rather than the check being made tolerant of its absence — a check that skips silently when an interface differs is the fail-open pattern this PR removes

What this does not claim

The warning does not stop a model fabricating; it makes fabrication visible when it reaches the user. And the opt-in eval tier's 19 cases have still never produced a complete run, so CONTRIBUTING's claim that LLM boundaries are "validated by the eval harness" remains half true — the CI tier is real, the model-facing tier is unproven.

Testing showed the prompt approach has stopped paying. An explicit instruction
added last change -- "aggregate statistics are an Execution tool, say so and
point the user there" -- was ignored in five analysis runs out of five. In the
worst run the agent made no query at all and invented both a log line
("ecutwfc=0.05 Ry") and the domain mechanics to justify it.

What has held all along is structural: requires_approval is never bypassed,
the router's strict parser refuses garbage, the fallback lands read-only. None
of those need the model to cooperate. So do that here instead.

1. query_run_context moves to the shared tools/ top level and is registered on
the Analysis agent too. It is read-only -- plain QueryBuilder queries -- and
sat on Execution only because Execution needed it first, not for any safety
reason. "What did my past runs use" is an exploration question by nature. With
both agents holding it, a misroute stops being a failure, which is already the
stated principle for search_aiida_docs and get_process_status. The situation is
removed rather than instructed against.

The prompt rule that was ignored is replaced by documentation of the tool,
including that reconstructing these statistics from query_nodes looks workable
and is not: the cutoff is not an extras field, so the query returns nulls, and
an answer assembled from nulls is a guess wearing the shape of a finding.

2. A grounding check now runs on every reply. aiida_agents.grounding pulls each
physical quantity out of the answer and reports any that appear in no tool
output, and the CLI prints a warning naming them. It catches fabrication
whether or not the model complied.

The detector lifts the eval harness's logic into the package and the harness
now imports it, so the check guarding a shipped answer and the check guarding
the test suite cannot drift -- two expressions of one idea drifting apart is
itself a bug this project has already had.

Writing its tests found the detector would have missed the very fabrication it
was built for. Requiring the number within ~20 characters of the parameter name
misses "ecutwfc varies widely (e.g., 42 for gold, 8 for the alkali metals)":
prose does not keep a value next to its name. It is now scoped to the sentence.
Unit spellings containing a digit ("1/A") had to stop being read as quantities
of their own -- a false positive is how a warning gets ignored.

Warns rather than blocks, and stays narrow: only numbers with a unit or in a
sentence naming a simulation parameter. "Found 12 structures" is prose.
@coderabbitai

coderabbitai Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds AiiDA run-context analytics to the Analysis agent and centralizes physical-quantity grounding checks. CLI commands and the REPL now warn about unsupported numerical claims, while tests cover query integration, grounding behavior, and tool evidence extraction.

Changes

Run-context analytics and grounding

Layer / File(s) Summary
Run-context query implementation
src/aiida_agents/tools/run_context.py, tests/tools/test_run_context.py
Adds structured ORM-backed queries for workflow history, available codes, failed attempts, and pseudopotential families.
Analysis-agent integration
src/aiida_agents/agents/analysis/..., src/aiida_agents/agents/execution/..., src/aiida_agents/tools/execution/..., tests/agents/...
Exposes query_run_context as an Analysis read-only tool, updates aggregate-statistics instructions, and redirects imports to the new module.
Grounding detection and CLI warnings
src/aiida_agents/grounding.py, src/aiida_agents/cli/...
Detects unsupported physical quantities using response text, tool outputs, and the user prompt, then warns in one-shot and REPL output.
Grounding validation alignment
tests/evals/_harness.py, tests/test_grounding.py, tests/cli/test_commands.py
Shares the runtime grounding implementation with evaluation checks and tests quantity extraction, normalization, evidence handling, and CLI compatibility.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AnalysisAgent
  participant query_run_context
  participant AiiDAORM
  AnalysisAgent->>query_run_context: submit query_type and filters
  query_run_context->>AiiDAORM: query AiiDA run-context records
  AiiDAORM-->>query_run_context: return ORM results
  query_run_context-->>AnalysisAgent: return structured statistics
Loading

Possibly related PRs

Suggested reviewers: geigerj2

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 86.21% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly reflects the main change: moving grounding from instructions to structural checks.
Description check ✅ Passed The description is detailed and clearly aligned with the grounding and shared tool changes in the PR.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jaweria-B
Jaweria-B merged commit ac90326 into main Jul 29, 2026
4 of 5 checks passed
@Jaweria-B
Jaweria-B deleted the feat/structural-grounding branch July 29, 2026 11:38
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details
putComment timed out

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 11

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiida_agents/agents/analysis/prompt.md`:
- Around line 72-73: Update the cross-reference in the introductory
aggregate-statistics guidance immediately before the “AGGREGATE STATISTICS OVER
PAST RUNS” section to point to rule 7 instead of rule 8; leave the surrounding
rules unchanged.

In `@src/aiida_agents/cli/output.py`:
- Around line 160-164: Update _warn_ungrounded so its diagnostic warning is
emitted to stderr rather than the shared Rich console stdout, preserving --raw
output as answer-only. Add or update the CLI test for _print_reply(raw=True) to
assert stdout contains only the <answer> content and excludes the
ungrounded-values warning.

In `@src/aiida_agents/grounding.py`:
- Line 46: Update the quantity extraction regex represented by _NUMBER to
tokenize optional leading plus or minus signs, ensuring values such as -5 eV
retain their sign and remain distinguishable from +5. Add a regression covering
opposite-sign evidence so warning suppression only occurs for matching signed
quantities.
- Line 121: Update the grounding calculation around quantities_in and
_numbers_in so evidence matching preserves parameter/unit context instead of
treating every numeric token as valid evidence. Exclude identifiers such as
process PKs, counts, and exit statuses from grounding, while retaining
tool-returned values that are explicitly associated with the relevant parameter
and unit; keep prompt-number subtraction behavior intact.
- Line 59: Update the _PARAM_SENTENCE regular expression in grounding.py to use
word boundaries around parameter names, preventing matches inside identifiers
such as ecutwfc_60. Add a regression test covering this identifier case and
verify that the numeric suffix is not reported as a physical claim.

In `@src/aiida_agents/tools/run_context.py`:
- Around line 270-271: Update the structure_type handling around workflow filter
construction and the response assignments to avoid defaulting or echoing
"metallic" when the caller omitted structure_type. Preserve the explicit
caller-provided structure_type filter, while leaving the existing
structure_type_filter_note behavior intact across the corresponding paths.
- Around line 364-391: Update the common_failure_modes construction in the
result-building flow to deduplicate failure_modes while preserving a
deterministic order, such as first occurrence order, before limiting the result
to three entries. Keep the existing failure_modes collection and output shape
unchanged.
- Around line 184-224: Remove the stale Analysis Agent framing from
query_run_context: revise the query_type Field description and function
docstring to describe querying run-context data directly via database-backed
context, without implying delegation or loose coupling between agents. Update
the related parameter and return documentation consistently while preserving the
existing query behavior.
- Around line 410-460: Update the available-codes result handling around the
codes_list guards: when a code_filter is provided and no rows match it, return
the filter-specific “no codes found matching” response regardless of the filter
text, while retaining the setup guidance only for unfiltered empty results. Also
remove the unreachable results-based conditional in the successful return and
use the appropriate always-successful note.
- Around line 491-511: Apply the structure_pk filter before limiting attempts to
10: iterate through the full records collection, retain only rows accepted by
the existing node structure check, and stop once 10 matching attempts have been
appended. Preserve the current attempt mapping and exception handling in the
surrounding attempts-building logic.
- Around line 470-490: Update the failed-attempts query in the QueryBuilder
filter construction to use AiiDA’s negated equality operator for
attributes.exit_status, replacing the current "!=" filter with the valid "!=="
form while preserving the existing nonzero-status behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d5fd8764-5d26-4120-85bd-244e8c3f415c

📥 Commits

Reviewing files that changed from the base of the PR and between a58ccc1 and 59aae7a.

📒 Files selected for processing (15)
  • src/aiida_agents/agents/analysis/__init__.py
  • src/aiida_agents/agents/analysis/prompt.md
  • src/aiida_agents/agents/execution/__init__.py
  • src/aiida_agents/cli/commands.py
  • src/aiida_agents/cli/output.py
  • src/aiida_agents/cli/repl.py
  • src/aiida_agents/grounding.py
  • src/aiida_agents/tools/execution/__init__.py
  • src/aiida_agents/tools/run_context.py
  • tests/agents/analysis/test_analysis.py
  • tests/agents/execution/test_integration.py
  • tests/cli/test_commands.py
  • tests/evals/_harness.py
  • tests/test_grounding.py
  • tests/tools/test_run_context.py

Comment on lines +72 to +73
- Aggregate statistics over past runs are yours too: see rule 8.
7. AGGREGATE STATISTICS OVER PAST RUNS:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Wrong rule number in cross-reference.

Line 72 says "see rule 8" but the aggregate-statistics guidance is rule 7 (the section immediately following, at line 73); rule 8 is "GROUNDING IN TOOL OUTPUT." This misdirects whatever reads this reference literally.

✏️ Suggested fix
-   - Aggregate statistics over past runs are yours too: see rule 8.
+   - Aggregate statistics over past runs are yours too: see rule 7.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- Aggregate statistics over past runs are yours too: see rule 8.
7. AGGREGATE STATISTICS OVER PAST RUNS:
- Aggregate statistics over past runs are yours too: see rule 7.
7. AGGREGATE STATISTICS OVER PAST RUNS:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/agents/analysis/prompt.md` around lines 72 - 73, Update the
cross-reference in the introductory aggregate-statistics guidance immediately
before the “AGGREGATE STATISTICS OVER PAST RUNS” section to point to rule 7
instead of rule 8; leave the surrounding rules unchanged.

Comment on lines +160 to +164
console.print(
f"[yellow]⚠ Not found in any tool output: {values}. "
"Verify before using these values.[/yellow]"
)
logger.warning("ungrounded quantities in reply: %s", values)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the relevant file and nearby code.
fd -a 'output\.py$' . | sed 's#^\./##'
echo '--- outline ---'
ast-grep outline src/aiida_agents/cli/output.py --view expanded 2>/dev/null || true
echo '--- relevant lines ---'
cat -n src/aiida_agents/cli/output.py | sed -n '1,230p'
echo '--- raw/console references ---'
rg -n "raw|console|_warn_ungrounded|ungrounded|stderr|stdout" src/aiida_agents/cli/output.py src -g '*.py' 2>/dev/null | head -200

Repository: aiidateam/aiida-agents

Length of output: 24424


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- Rich Console stdout implementation ---'
python3 - <<'PY'
try:
    from rich.console import Console
    import io
except Exception as exc:
    print(type(exc).__name__, exc)
    raise SystemExit(0)

buf = io.StringIO()
console = Console(file=buf)
console.print("x")
print("file:", console.file)
print("stdout:", console.file == console.stdout)
print("buf after first:", buf.getvalue(), repr(buf.getvalue()))
console.print("y", end="\n\n")
print("stdout:", console.file == console.stdout)
print("buf after second:", buf.getvalue())
PY

echo '--- CLI command relevant lines ---'
cat -n src/aiida_agents/cli/commands.py | sed -n '90,155p'
echo '--- tests for raw/uncorrupted stdout ---'
rg -n "raw|stdout|answer\.md|pipe|warn|ungrounded" tests -g '*.py' | head -200

Repository: aiidateam/aiida-agents

Length of output: 10351


Keep --raw stdout free of diagnostics.

_print_reply(raw=raw) writes only <answer> to stdout, but _warn_ungrounded writes ⚠ ... via the shared Rich console after it, so aiida-agents ask ... --raw can append warning text to the captured Markdown. Emit the warning on stderr and add a --raw stdout assertion.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/cli/output.py` around lines 160 - 164, Update
_warn_ungrounded so its diagnostic warning is emitted to stderr rather than the
shared Rich console stdout, preserving --raw output as answer-only. Add or
update the CLI test for _print_reply(raw=True) to assert stdout contains only
the <answer> content and excludes the ungrounded-values warning.

r"press_conv_thr|nbnd|electron_maxstep|cutoff"
)

_NUMBER = r"\d+(?:\.\d+)?(?:[eE][-+]?\d+)?"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve the quantity sign.

-5 eV is extracted as 5.0, so evidence containing +5 incorrectly suppresses the warning. Tokenize signed literals and add a regression for opposite-sign evidence.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 46, Update the quantity extraction
regex represented by _NUMBER to tokenize optional leading plus or minus signs,
ensuring values such as -5 eV retain their sign and remain distinguishable from
+5. Add a regression covering opposite-sign evidence so warning suppression only
occurs for matching signed quantities.

#: metals)" puts 22 characters between them, and the 8 much further. Prose does
#: not keep a value next to its name. Scoping to the sentence matches how the
#: claim is actually written.
_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Bound parameter names to avoid identifier false positives.

ecutwfc_60 matches ecutwfc, then the sentence scan reports 60 as a physical claim. This contradicts the stated identifier exclusion. Add word boundaries and a regression case.

Proposed fix
-_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)
+_PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)
_PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 59, Update the _PARAM_SENTENCE regular
expression in grounding.py to use word boundaries around parameter names,
preventing matches inside identifiers such as ecutwfc_60. Add a regression test
covering this identifier case and verify that the numeric suffix is not reported
as a physical claim.

Normalised numeric literals with no source. Empty means every physics
number in the answer is traceable to a tool or to the question.
"""
return quantities_in(answer) - _numbers_in(evidence) - _numbers_in(prompt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Match evidence claims, not arbitrary numeric tokens.

A tool return such as Process pk 60 finished makes Use ecutwfc 60 Ry appear grounded. PKs, counts, and exit statuses can therefore bless fabricated parameters. Preserve parameter/unit context when extracting evidence, rather than subtracting every number in the tool text.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 121, Update the grounding calculation
around quantities_in and _numbers_in so evidence matching preserves
parameter/unit context instead of treating every numeric token as valid
evidence. Exclude identifiers such as process PKs, counts, and exit statuses
from grounding, while retaining tool-returned values that are explicitly
associated with the relevant parameter and unit; keep prompt-number subtraction
behavior intact.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 11

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiida_agents/agents/analysis/prompt.md`:
- Around line 72-73: Update the cross-reference in the introductory
aggregate-statistics guidance immediately before the “AGGREGATE STATISTICS OVER
PAST RUNS” section to point to rule 7 instead of rule 8; leave the surrounding
rules unchanged.

In `@src/aiida_agents/cli/output.py`:
- Around line 160-164: Update _warn_ungrounded so its diagnostic warning is
emitted to stderr rather than the shared Rich console stdout, preserving --raw
output as answer-only. Add or update the CLI test for _print_reply(raw=True) to
assert stdout contains only the <answer> content and excludes the
ungrounded-values warning.

In `@src/aiida_agents/grounding.py`:
- Line 46: Update the quantity extraction regex represented by _NUMBER to
tokenize optional leading plus or minus signs, ensuring values such as -5 eV
retain their sign and remain distinguishable from +5. Add a regression covering
opposite-sign evidence so warning suppression only occurs for matching signed
quantities.
- Line 121: Update the grounding calculation around quantities_in and
_numbers_in so evidence matching preserves parameter/unit context instead of
treating every numeric token as valid evidence. Exclude identifiers such as
process PKs, counts, and exit statuses from grounding, while retaining
tool-returned values that are explicitly associated with the relevant parameter
and unit; keep prompt-number subtraction behavior intact.
- Line 59: Update the _PARAM_SENTENCE regular expression in grounding.py to use
word boundaries around parameter names, preventing matches inside identifiers
such as ecutwfc_60. Add a regression test covering this identifier case and
verify that the numeric suffix is not reported as a physical claim.

In `@src/aiida_agents/tools/run_context.py`:
- Around line 270-271: Update the structure_type handling around workflow filter
construction and the response assignments to avoid defaulting or echoing
"metallic" when the caller omitted structure_type. Preserve the explicit
caller-provided structure_type filter, while leaving the existing
structure_type_filter_note behavior intact across the corresponding paths.
- Around line 364-391: Update the common_failure_modes construction in the
result-building flow to deduplicate failure_modes while preserving a
deterministic order, such as first occurrence order, before limiting the result
to three entries. Keep the existing failure_modes collection and output shape
unchanged.
- Around line 184-224: Remove the stale Analysis Agent framing from
query_run_context: revise the query_type Field description and function
docstring to describe querying run-context data directly via database-backed
context, without implying delegation or loose coupling between agents. Update
the related parameter and return documentation consistently while preserving the
existing query behavior.
- Around line 410-460: Update the available-codes result handling around the
codes_list guards: when a code_filter is provided and no rows match it, return
the filter-specific “no codes found matching” response regardless of the filter
text, while retaining the setup guidance only for unfiltered empty results. Also
remove the unreachable results-based conditional in the successful return and
use the appropriate always-successful note.
- Around line 491-511: Apply the structure_pk filter before limiting attempts to
10: iterate through the full records collection, retain only rows accepted by
the existing node structure check, and stop once 10 matching attempts have been
appended. Preserve the current attempt mapping and exception handling in the
surrounding attempts-building logic.
- Around line 470-490: Update the failed-attempts query in the QueryBuilder
filter construction to use AiiDA’s negated equality operator for
attributes.exit_status, replacing the current "!=" filter with the valid "!=="
form while preserving the existing nonzero-status behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d5fd8764-5d26-4120-85bd-244e8c3f415c

📥 Commits

Reviewing files that changed from the base of the PR and between a58ccc1 and 59aae7a.

📒 Files selected for processing (15)
  • src/aiida_agents/agents/analysis/__init__.py
  • src/aiida_agents/agents/analysis/prompt.md
  • src/aiida_agents/agents/execution/__init__.py
  • src/aiida_agents/cli/commands.py
  • src/aiida_agents/cli/output.py
  • src/aiida_agents/cli/repl.py
  • src/aiida_agents/grounding.py
  • src/aiida_agents/tools/execution/__init__.py
  • src/aiida_agents/tools/run_context.py
  • tests/agents/analysis/test_analysis.py
  • tests/agents/execution/test_integration.py
  • tests/cli/test_commands.py
  • tests/evals/_harness.py
  • tests/test_grounding.py
  • tests/tools/test_run_context.py
🛑 Comments failed to post (6)
src/aiida_agents/tools/run_context.py (6)

184-224: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Stale "asks the Analysis Agent" framing in the tool's own docs.

The module docstring (lines 9-15) explicitly says this tool was renamed away from query_analysis_agent because a docstring claiming it "asked the Analysis agent" was never true — it runs QueryBuilder queries directly. But the Field description ("What to ask Analysis Agent...") and this function's own docstring ("Query the Analysis Agent for context...loose coupling between Execution Agent and Analysis Agent...instead of Execution Agent importing Analysis Agent's tools directly") still carry that exact framing. This is now also registered directly on the Analysis agent's own toolset, so the model reading "what to ask Analysis Agent" while being the Analysis agent itself is genuinely confusing — the same failure mode the rename was meant to fix.

✏️ Suggested doc fix
     query_type: t.Annotated[
         str,
         Field(
             description=(
-                "What to ask Analysis Agent. Options: "
+                "What run-context statistic to retrieve from the active AiiDA "
+                "profile. Options: "
                 "'past_successful_workflows', 'available_codes', "
                 "'failed_attempts', 'available_pseudos'"
             )
         ),
     ],
-    """Query the Analysis Agent for context before generating a workflow spec.
-
-    This tool provides loose coupling between Execution Agent and Analysis Agent.
-    Instead of Execution Agent importing Analysis Agent's tools directly, it calls
-    query_run_context() and gets back structured context.
-
-    The Execution Agent uses this context to:
+    """Query the active AiiDA profile for run-context statistics.
+
+    Both the Analysis and Execution agents call this directly (it runs
+    ``QueryBuilder`` queries in-process; see the module docstring). Callers
+    use this context to:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

def query_run_context(
    query_type: t.Annotated[
        str,
        Field(
            description=(
                "What run-context statistic to retrieve from the active AiiDA "
                "profile. Options: "
                "'past_successful_workflows', 'available_codes', "
                "'failed_attempts', 'available_pseudos'"
            )
        ),
    ],
    filters: t.Annotated[
        dict[str, t.Any],
        Field(
            description=(
                "Context-specific filters (e.g. structure_type, composition, workflow_type). "
                "Note: 'structure_type' is not a database-level filter — AiiDA stores no "
                "per-node material-class attribute. It is echoed back as metadata only."
            )
        ),
    ],
) -> dict[str, t.Any]:
    """Query the active AiiDA profile for run-context statistics.

    Both the Analysis and Execution agents call this directly (it runs
    ``QueryBuilder`` queries in-process; see the module docstring). Callers
    use this context to:
    1. Learn what parameters worked for similar structures in the past.
    2. Check what codes are available.
    3. Understand why similar setups failed.
    4. Make better parameter choices.

    Args:
        query_type: The kind of information to retrieve.
        filters: Context that narrows the query (workflow_type, structure_type, etc.).

    Returns:
        dict with context-specific data (past runs, codes, failures, etc.).
    """
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 184 - 224, Remove the
stale Analysis Agent framing from query_run_context: revise the query_type Field
description and function docstring to describe querying run-context data
directly via database-backed context, without implying delegation or loose
coupling between agents. Update the related parameter and return documentation
consistently while preserving the existing query behavior.

270-271: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Unrequested structure_type default is echoed back as if it were the real filter.

structure_type defaults to "metallic" when the caller omits it, then that default is echoed verbatim into res["structure_type"] even though the accompanying structure_type_filter_note already says it isn't a real filter. A caller that didn't ask about metals will still see "structure_type": "metallic" in the response, which is exactly the kind of unsourced-looking value this PR's grounding effort is trying to eliminate.

✏️ Suggested fix
-    structure_type = filters.get("structure_type", "metallic")
+    structure_type = filters.get("structure_type")

Also applies to: 293-294, 378-379

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 270 - 271, Update the
structure_type handling around workflow filter construction and the response
assignments to avoid defaulting or echoing "metallic" when the caller omitted
structure_type. Preserve the explicit caller-provided structure_type filter,
while leaving the existing structure_type_filter_note behavior intact across the
corresponding paths.

364-391: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

common_failure_modes ordering is nondeterministic across process runs.

list(set(failure_modes))[:3] dedupes via a set, whose string iteration order depends on Python's per-process hash seed (randomized by default). Two identical databases queried from two separate process launches can return different common_failure_modes, undermining reproducibility of a tool whose whole point is to be a trustworthy source of truth.

✏️ Suggested fix
-        "common_failure_modes": list(set(failure_modes))[:3],
+        "common_failure_modes": list(dict.fromkeys(failure_modes))[:3],
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    failure_modes: list[str] = []
    for row in failed_runs[:10]:
        try:
            fnode = orm.load_node(row[1])
            if fnode.exit_message:
                failure_modes.append(f"Exit status {row[0]}: {fnode.exit_message}")
            else:
                failure_modes.append(f"Exit status {row[0]}")
        except Exception as exc:
            logger.debug("Could not load failed node %s: %s", row[1], exc)

    return {
        "query_type": "past_successful_workflows",
        "workflow_type": workflow_type,
        "structure_type": structure_type,
        "structure_type_filter_note": (
            "structure_type is not filterable at the database level; "
            "all finished workflows of this type are included."
        ),
        "count": len(successful_runs),
        "success_rate": success_rate,
        "median_ecutwfc": med_ecutwfc,
        "median_kpoints_distance": med_kpoints,
        "units": _UNITS,
        "common_parameters": {"ecutwfc": med_ecutwfc} if med_ecutwfc else {},
        "common_failure_modes": list(dict.fromkeys(failure_modes))[:3],
        "example_structures": example_structs,
        "note": f"Analyzed {total_runs} historical workflow(s) in active database ({len(successful_runs)} successful).",
🧰 Tools
🪛 Ruff (0.16.0)

[warning] 372-372: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 364 - 391, Update the
common_failure_modes construction in the result-building flow to deduplicate
failure_modes while preserving a deterministic order, such as first occurrence
order, before limiting the result to three entries. Keep the existing
failure_modes collection and output shape unchanged.

410-460: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

"No codes matching filter" message only fires when the filter string literally contains "unknown".

The branch at line 432 that reports f"No codes found matching {code_filter!r}" is gated on code_filter and "unknown" in str(code_filter).lower() — a heuristic that only matches the test fixture's literal "unknown-abinit-code" string. A real filter like "abinit-v2" that matches nothing falls through to the generic 440-449 branch and tells the caller "No codes found in the active AiiDA profile. Please set up a code first," which is false when codes exist but just didn't match the filter — a misleading answer that could send the agent down the wrong path (e.g. telling the user to install a code that's already installed under a different label). Separately, the if results else ... at line 457 is unreachable dead code: by that point codes_list (and hence results, its superset) is already known non-empty from the two guards above.

🐛 Suggested fix
-    if not codes_list and code_filter and "unknown" in str(code_filter).lower():
-        return {
-            "query_type": "available_codes",
-            "codes": [],
-            "recommended_version": None,
-            "note": f"No codes found matching {code_filter!r} in active AiiDA profile.",
-        }
-
-    if not codes_list:
+    if not codes_list:
+        if code_filter:
+            note = f"No codes found matching {code_filter!r} in active AiiDA profile."
+        else:
+            note = (
+                "No codes found in the active AiiDA profile. "
+                "Please set up a code first: verdi code setup"
+            )
         return {
             "query_type": "available_codes",
             "codes": [],
             "recommended_version": None,
-            "note": (
-                "No codes found in the active AiiDA profile. "
-                "Please set up a code first: verdi code setup"
-            ),
+            "note": note,
         }

     return {
         "query_type": "available_codes",
         "codes": codes_list,
         "recommended_version": recommended,
-        "note": (
-            f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile."
-            if results
-            else "No matching codes found in active AiiDA profile. Providing schema recommendation."
-        ),
+        "note": f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile.",
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    code_filter = (
        filters.get("code") or filters.get("plugin") or filters.get("code_label")
    )
    codes_list: list[dict[str, str]] = []
    recommended: str | None = None
    for row in results:
        label, plugin, desc = row[0], row[1], row[2]
        if code_filter and (
            str(code_filter).lower() not in str(label).lower()
            and str(code_filter).lower() not in str(plugin).lower()
        ):
            continue
        codes_list.append(
            {
                "label": str(label),
                "plugin": str(plugin) if plugin else "unknown",
                "description": str(desc) if desc else "",
            }
        )
        if not recommended or "pw" in str(label).lower():
            recommended = str(label)

    if not codes_list:
        if code_filter:
            note = f"No codes found matching {code_filter!r} in active AiiDA profile."
        else:
            note = (
                "No codes found in the active AiiDA profile. "
                "Please set up a code first: verdi code setup"
            )
        return {
            "query_type": "available_codes",
            "codes": [],
            "recommended_version": None,
            "note": note,
        }

    return {
        "query_type": "available_codes",
        "codes": codes_list,
        "recommended_version": recommended,
        "note": f"Found {len(codes_list)} available code(s) matching filter in active AiiDA profile.",
    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 410 - 460, Update the
available-codes result handling around the codes_list guards: when a code_filter
is provided and no rows match it, return the filter-specific “no codes found
matching” response regardless of the filter text, while retaining the setup
guidance only for unfiltered empty results. Also remove the unreachable
results-based conditional in the successful return and use the appropriate
always-successful note.

470-490: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Does AiiDA's orm.QueryBuilder support "!=" as a filter operator, or only "!=="?

💡 Result:

AiiDA's QueryBuilder supports the inequality operator by prepending an exclamation mark (!) to the equality operator (==), resulting in!== [1][2][3]. While you can use the!== syntax to explicitly represent "not equal to," the QueryBuilder's filter system is designed so that any supported operator can be negated by adding a! prefix [1][3]. In practice,!== is the standard way to express "not equal" for the equality operator, which the QueryBuilder subsequently translates into a standard!= SQL operator [1][3]. Additionally, as of AiiDA 2.6, a new programmatic filter negation syntax was introduced allowing for the use of the ~ operator (e.g., ~(field == value)), which provides an alternative way to express negation [2].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file excerpt =="
wc -l src/aiida_agents/tools/run_context.py
sed -n '440,510p' src/aiida_agents/tools/run_context.py

echo
echo "== dependency/version hints =="
git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|Pipfile|poetry\.lock|uv\.lock|setup\.py|setup\.cfg)$' || true
for f in $(git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|Pipfile|poetry\.lock|uv\.lock|setup\.py|setup\.cfg)$' || true); do
  echo "--- $f ---"
  rg -n "aiida|aiida-core|alembic|pydantic|pytest" "$f" || true
done

echo
echo "== prompt guidance occurrence =="
rg -n 'Not-equal is|!=|!==|QueryBuilder|failed attempts' -S .

Repository: aiidateam/aiida-agents

Length of output: 50378


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== tests mentioning failed_attempts / query_failed_attempts =="
rg -n "failed_attempts|query_failed_attempts|failed attempts|exit_status|exit_status|attempts" tests src -g 'test*.py' -g '*.py' || true

echo
echo "== relevant prompt / guidance =="
rg -n -C 3 'Not-equal is|!=|!==|QueryBuilder|failed attempts' prompt.md CONTRIBUTING.md src/aiida_agents -g '!*.pyc' || true

echo
echo "== locate QueryBuilder operator validation within source / lock =="
tmpdir="$(mktemp -d)"
cd "$tmpdir"
git clone --depth 1 --branch v3.9.0 https://github.com/aiidateam/aiida-core.git >/tmp/clone.log 2>&1 || true
if [ -d aiida-core ]; then
  cd aiida-core
  rg -n "!==|!=".*"?=|OPERATOR|operators|validate_operator|QueryBuilder" aiida -g '*.py' | head -n 200 || true
else
  echo "aiida-core checkout unavailable"
fi

Repository: aiidateam/aiida-agents

Length of output: 152


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== local test and prompt references (fast, targeted) =="
python3 /bin/bash - <<'PY'
import subprocess
result = subprocess.run(
    ["rg", "-n", "-C", "2", "query_failed_attempts|failed_attempts|Not-equal is|Not equal|Not equal|!=.*:|!==|QueryBuilderDict|QueryBuilder"],
    "--glob", "**/*.py",
    "--glob", "**/*.md",
    pytest,
    capture_output=True,
    text=True,
    timeout=60
)
PY

echo
echo "== inspect query_failed_attempts call sites only =="
rg -n "query_failed_attempts" src tests -g '*.py' || true

Repository: aiidateam/aiida-agents

Length of output: 374


🌐 Web query:

AiiDA QueryBuilder negation operator '!=' unsupported '=='

💡 Result:

In AiiDA QueryBuilder, the != operator is not directly supported as a standalone operator. Instead, to perform a negation (such as "not equal to"), you must prepend an exclamation mark ! to the corresponding operator [1][2]. To express "not equal to," you use the negation of the equality operator ==, which is written as !== [1][2][3]. Example usage: filters={'some_field': {'!==': 'value_to_exclude'}} Additionally, for newer versions of AiiDA (2.6 and later), a logical expression syntax allows for the use of the ~ operator to negate any expression, such as ~ (Node.fields.some_field == 'value') [1][2][3]. You can also use ~== in some contexts to represent inequality [4].

Citations:


Use AiiDA’s inequality operator for failed_attempts.

{"!=": 0} is not a valid QueryBuilder filter operator; write the negated equality operator as {"!==": 0}. Otherwise the query raises, the surrounding except Exception swallows it, and the tool reports no failed attempts even when failed attempts exist.

🧰 Tools
🪛 Ruff (0.16.0)

[warning] 487-487: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 470 - 490, Update the
failed-attempts query in the QueryBuilder filter construction to use AiiDA’s
negated equality operator for attributes.exit_status, replacing the current "!="
filter with the valid "!==" form while preserving the existing nonzero-status
behavior.

491-511: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

structure_pk post-filter is applied after truncating to the first 10 raw records.

for row in records[:10]: slices the unfiltered query results to 10 rows before the structure_pk match check runs. If a database has more than 10 failed attempts and the ones matching structure_pk are not among the first 10 (arbitrary DB order, no order_by), they are never inspected — the caller can get zero or fewer than 10 matching attempts even though more exist. This defeats the purpose of the structure_pk filter for any database with a non-trivial number of failures.

🐛 Suggested fix
-    attempts: list[dict[str, t.Any]] = []
-    for row in records[:10]:
+    attempts: list[dict[str, t.Any]] = []
+    for row in records:
+        if len(attempts) >= 10:
+            break
         pk, status, msg = row[0], row[1], row[2]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    attempts: list[dict[str, t.Any]] = []
    for row in records:
        if len(attempts) >= 10:
            break
        pk, status, msg = row[0], row[1], row[2]
        if structure_pk:
            try:
                node = orm.load_node(pk)
                if "structure" in node.inputs and str(node.inputs.structure.pk) != str(
                    structure_pk
                ):
                    continue
            except Exception as exc:
                logger.debug("Could not load node %s for structure filter: %s", pk, exc)
                continue
        attempts.append(
            {
                "pk": pk,
                "exit_code": status,
                "failure_reason": msg or f"Failed with exit status {status}",
            }
        )
🧰 Tools
🪛 Ruff (0.16.0)

[warning] 501-501: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/run_context.py` around lines 491 - 511, Apply the
structure_pk filter before limiting attempts to 10: iterate through the full
records collection, retain only rows accepted by the existing node structure
check, and stop once 10 matching attempts have been appended. Preserve the
current attempt mapping and exception handling in the surrounding
attempts-building logic.

@Jaweria-B Jaweria-B changed the title ✨ make grounding structural instead of instructed ✨ make grounding structural, and stop the tools asserting what they cannot support Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant