Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/aiida_agents/agents/analysis/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
list_processes,
list_retrieved_files,
query_nodes,
query_run_context,
search_structures,
)
from aiida_agents.rag import search_aiida_docs
Expand All @@ -54,6 +55,7 @@
get_node_inputs,
get_node_outputs,
search_structures,
query_run_context,
search_aiida_docs,
]

Expand Down
22 changes: 15 additions & 7 deletions src/aiida_agents/agents/analysis/prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,21 @@ CRITICAL TOOL SELECTION RULES:
started, queued, or submitted anything.
- You can still fully answer questions *about* a workflow — what it does, what inputs it
takes — using your read tools and 'search_aiida_docs'.
- Aggregate statistics over past runs (typical ecutwfc, kpoints_distance, success rates,
common failure modes) come from 'query_run_context', which is an Execution agent tool and
is NOT one of yours. If asked for them, say so and point the user at the execution agent
('aiida-agents -a execution ask ...'). You may report what individual nodes contain via
'query_nodes' and 'get_node_inputs', reporting only values those tools actually returned.
Do not reconstruct a statistic you have no tool for by estimating it.
7. GROUNDING IN TOOL OUTPUT:
- Aggregate statistics over past runs are yours too: see rule 8.
7. AGGREGATE STATISTICS OVER PAST RUNS:
Comment on lines +72 to +73

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Wrong rule number in cross-reference.

Line 72 says "see rule 8" but the aggregate-statistics guidance is rule 7 (the section immediately following, at line 73); rule 8 is "GROUNDING IN TOOL OUTPUT." This misdirects whatever reads this reference literally.

✏️ Suggested fix
-   - Aggregate statistics over past runs are yours too: see rule 8.
+   - Aggregate statistics over past runs are yours too: see rule 7.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- Aggregate statistics over past runs are yours too: see rule 8.
7. AGGREGATE STATISTICS OVER PAST RUNS:
- Aggregate statistics over past runs are yours too: see rule 7.
7. AGGREGATE STATISTICS OVER PAST RUNS:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/agents/analysis/prompt.md` around lines 72 - 73, Update the
cross-reference in the introductory aggregate-statistics guidance immediately
before the “AGGREGATE STATISTICS OVER PAST RUNS” section to point to rule 7
instead of rule 8; leave the surrounding rules unchanged.

- For "what ecutwfc did my successful relaxations use", "how often does this workflow succeed",
"what usually goes wrong with it", use 'query_run_context(query_type=..., filters=...)'.
query_type is one of 'past_successful_workflows', 'available_codes', 'failed_attempts',
'available_pseudos'.
- It reads nested workflow inputs correctly (a cutoff lives at base.pw.parameters on a
PwRelaxWorkChain, not at the top level) and returns a 'units' field. Quote those units
verbatim; do not supply your own.
- Do NOT reconstruct these statistics yourself out of 'query_nodes' results. That path looks
workable and is not: the cutoff is not an extras field, so the query returns nulls, and an
answer assembled from nulls is a guess wearing the shape of a finding.
- If 'query_run_context' returns nulls or a zero count, say exactly that. "The database records
no ecutwfc for these runs" is a complete answer.
8. GROUNDING IN TOOL OUTPUT:
- Prefer retrieved code exactly as shown.
- If retrieved code doesn't apply directly, you may adapt it minimally (e.g., changing variable names
for clarity), but explain the adaptation and keep the core logic unchanged.
Expand Down
2 changes: 1 addition & 1 deletion src/aiida_agents/agents/execution/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from aiida_agents.agents._errors import RetryOnToolError
from aiida_agents.agents._models import get_model

from aiida_agents.tools.execution.run_context import query_run_context
from aiida_agents.tools.run_context import query_run_context
from aiida_agents.tools.execution.codes import list_codes
from aiida_agents.tools.execution.introspection import describe_workflow, list_workflows
from aiida_agents.tools.execution.protocol import build_workflow_inputs
Expand Down
2 changes: 2 additions & 0 deletions src/aiida_agents/cli/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from aiida_agents.cli.doctor import doctor
from aiida_agents.cli.mcp import mcp
from aiida_agents.cli.output import (
_warn_ungrounded,
_format_duration,
_print_reply,
_render_tool_calls,
Expand Down Expand Up @@ -143,6 +144,7 @@ def ask_cmd(ctx: click.Context, question: str, raw: bool) -> None:
)
raise SystemExit(2)
_print_reply(result.output, raw=raw)
_warn_ungrounded(result.output, result.all_messages(), question)


@cli.command()
Expand Down
33 changes: 33 additions & 0 deletions src/aiida_agents/cli/output.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

from aiida_agents._logging import ToolPart, trace_response, trace_tool_part

logger = logging.getLogger(__name__)

console = Console()


Expand Down Expand Up @@ -129,3 +131,34 @@ def _format_duration(seconds: float) -> str:
return f"{seconds:.1f}s"
minutes, secs = divmod(int(seconds), 60)
return f"{minutes}m {secs}s"


def _warn_ungrounded(text: str, messages: list[ModelMessage], question: str) -> None:
"""Flag any physical quantity in a reply that no tool produced.

The agents are asked what cutoff to use, so a wrong number is not a wrong
answer -- it configures a calculation. The failure that occurs is a
plausible value, often attached to a real label the model did retrieve, and
the user cannot tell it apart from a sourced one by reading.

This runs on every reply because the prompt-level version of the same rule
has a measured failure rate: an explicit instruction not to do this was
ignored in five test runs out of five. A check that reads the answer
afterwards does not depend on the model having complied.

Warns rather than blocks: the detector is deliberately narrow (see
``aiida_agents.grounding``) but a false positive must cost a line of output,
never a withheld answer.
"""
from aiida_agents.grounding import tool_output_text, ungrounded_quantities

invented = ungrounded_quantities(text, tool_output_text(messages), question)
if not invented:
return

values = ", ".join(sorted(invented))
console.print(
f"[yellow]⚠ Not found in any tool output: {values}. "
"Verify before using these values.[/yellow]"
)
logger.warning("ungrounded quantities in reply: %s", values)
Comment on lines +160 to +164

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the relevant file and nearby code.
fd -a 'output\.py$' . | sed 's#^\./##'
echo '--- outline ---'
ast-grep outline src/aiida_agents/cli/output.py --view expanded 2>/dev/null || true
echo '--- relevant lines ---'
cat -n src/aiida_agents/cli/output.py | sed -n '1,230p'
echo '--- raw/console references ---'
rg -n "raw|console|_warn_ungrounded|ungrounded|stderr|stdout" src/aiida_agents/cli/output.py src -g '*.py' 2>/dev/null | head -200

Repository: aiidateam/aiida-agents

Length of output: 24424


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- Rich Console stdout implementation ---'
python3 - <<'PY'
try:
    from rich.console import Console
    import io
except Exception as exc:
    print(type(exc).__name__, exc)
    raise SystemExit(0)

buf = io.StringIO()
console = Console(file=buf)
console.print("x")
print("file:", console.file)
print("stdout:", console.file == console.stdout)
print("buf after first:", buf.getvalue(), repr(buf.getvalue()))
console.print("y", end="\n\n")
print("stdout:", console.file == console.stdout)
print("buf after second:", buf.getvalue())
PY

echo '--- CLI command relevant lines ---'
cat -n src/aiida_agents/cli/commands.py | sed -n '90,155p'
echo '--- tests for raw/uncorrupted stdout ---'
rg -n "raw|stdout|answer\.md|pipe|warn|ungrounded" tests -g '*.py' | head -200

Repository: aiidateam/aiida-agents

Length of output: 10351


Keep --raw stdout free of diagnostics.

_print_reply(raw=raw) writes only <answer> to stdout, but _warn_ungrounded writes ⚠ ... via the shared Rich console after it, so aiida-agents ask ... --raw can append warning text to the captured Markdown. Emit the warning on stderr and add a --raw stdout assertion.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/cli/output.py` around lines 160 - 164, Update
_warn_ungrounded so its diagnostic warning is emitted to stderr rather than the
shared Rich console stdout, preserving --raw output as answer-only. Add or
update the CLI test for _print_reply(raw=True) to assert stdout contains only
the <answer> content and excludes the ungrounded-values warning.

2 changes: 2 additions & 0 deletions src/aiida_agents/cli/repl.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from aiida_agents.cli.output import (
_format_duration,
_print_agent,
_warn_ungrounded,
_render_tool_calls,
console,
)
Expand Down Expand Up @@ -156,6 +157,7 @@ def _run_turn(
history = _handle_deferred(agent, result, history)
else:
_print_agent(result.output)
_warn_ungrounded(result.output, result.all_messages(), question)
history = result.all_messages()
console.print(f"[dim]⏱ {_format_duration(elapsed)}[/]")
return history
Expand Down
141 changes: 141 additions & 0 deletions src/aiida_agents/grounding.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
"""Is a number in an answer traceable to something a tool returned?

The agents are asked what cutoff to use, and a wrong answer is not an
inconvenience -- it configures a calculation that burns real CPU hours. The
failure that actually occurs is not a refusal or an obvious error: it is a
plausible number, often attached to a real label the model did retrieve
("42 for gold", where the gold came from a query and the 42 came from
nowhere). That reads as sourced and cannot be checked by eye.

Prompt rules asking the model not to do this have a measurable failure rate:
across one session's testing, an explicit instruction was ignored in five runs
out of five, and fabrication recurred intermittently on every model tried. So
this check does not ask. It reads the answer after the fact and reports which
quantities no tool produced, which works whether or not the model cooperated.

Deliberately narrow. Only numbers carrying a unit, or bound to a named
simulation parameter, count as claims about physics -- "step 2 of 3" and
"found 12 structures" are prose. Values are normalised, so 60 in the answer
matches 60.0 in a tool return. A check that cried wolf would be switched off,
and then it would protect nothing.
"""

from __future__ import annotations

import re
from typing import Any

__all__ = ["quantities_in", "tool_output_text", "ungrounded_quantities"]


#: Units whose presence makes the preceding number a physics claim. ASCII
#: spellings are included because tool output and terminals both use them --
#: ``query_run_context`` reports "1/A", not "1/Angstrom-with-a-ring".
_UNITS = (
r"Ry|Rydberg|eV|meV|Ha|Hartree|Bohr|bohr|GPa|kbar|K"
r"|Å\^?-1|Å⁻¹|1/Å|Å|A\^-1|A-1|1/A|Ang|angstrom"
)

#: Parameter names whose values are physics claims regardless of unit.
_PARAMS = (
r"ecutwfc|ecutrho|conv_thr|degauss|kpoints?_distance|"
r"k-?point spacing|mixing_beta|smearing|etot_conv_thr|forc_conv_thr|"
r"press_conv_thr|nbnd|electron_maxstep|cutoff"
)

_NUMBER = r"\d+(?:\.\d+)?(?:[eE][-+]?\d+)?"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Preserve the quantity sign.

-5 eV is extracted as 5.0, so evidence containing +5 incorrectly suppresses the warning. Tokenize signed literals and add a regression for opposite-sign evidence.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 46, Update the quantity extraction
regex represented by _NUMBER to tokenize optional leading plus or minus signs,
ensuring values such as -5 eV retain their sign and remain distinguishable from
+5. Add a regression covering opposite-sign evidence so warning suppression only
occurs for matching signed quantities.


#: A number carrying a unit -- "60 Ry", "0.15 1/A" -- anywhere in the text.
_WITH_UNIT = re.compile(rf"(?P<num>{_NUMBER})\s*(?:{_UNITS})\b")

#: A sentence that names a simulation parameter.
#:
#: Proximity was the first attempt and was wrong: requiring the number within
#: ~20 characters of the parameter name missed the fabrication this check was
#: built for -- "ecutwfc varies widely (e.g., 42 for gold, 8 for the alkali
#: metals)" puts 22 characters between them, and the 8 much further. Prose does
#: not keep a value next to its name. Scoping to the sentence matches how the
#: claim is actually written.
_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Bound parameter names to avoid identifier false positives.

ecutwfc_60 matches ecutwfc, then the sentence scan reports 60 as a physical claim. This contradicts the stated identifier exclusion. Add word boundaries and a regression case.

Proposed fix
-_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)
+_PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
_PARAM_SENTENCE = re.compile(rf"(?:{_PARAMS})", re.IGNORECASE)
_PARAM_SENTENCE = re.compile(rf"\b(?:{_PARAMS})\b", re.IGNORECASE)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 59, Update the _PARAM_SENTENCE regular
expression in grounding.py to use word boundaries around parameter names,
preventing matches inside identifiers such as ecutwfc_60. Add a regression test
covering this identifier case and verify that the numeric suffix is not reported
as a physical claim.


#: A unit written on its own, for removal before counting bare numbers.
_UNIT_TOKEN = re.compile(rf"(?:{_UNITS})")
_SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+|\n")


def _numbers_in(text: str) -> set[str]:
"""Every numeric literal in ``text``, normalised for comparison."""
return {_normalise(raw) for raw in re.findall(_NUMBER, text)}


def _normalise(number: str) -> str:
"""Canonicalise a numeric literal so 60, 60.0 and 6e1 compare equal.

Without this the check reports false fabrications constantly: a tool
returns ``60.0`` and the model writes "60 Ry", which is the same claim
correctly repeated.
"""
try:
return repr(float(number))
except ValueError: # pragma: no cover - regex only yields parseable numbers
return number


def quantities_in(text: str) -> set[str]:
"""Physical quantities asserted in ``text``, as normalised numbers.

A number counts when it carries a unit, or when it appears in a sentence
naming a simulation parameter. Bare numbers in ordinary prose ("step 2 of
3", "found 12 structures") do not: flagging those would bury the signal,
and a warning nobody believes protects nothing.
"""
found = {_normalise(m.group("num")) for m in _WITH_UNIT.finditer(text)}

for sentence in _SENTENCE_SPLIT.split(text):
if not _PARAM_SENTENCE.search(sentence):
continue
# Strip the number+unit pairs already collected above, then any bare
# unit token, before scanning for unattached numbers. Some unit
# spellings contain a digit of their own -- "1/A", "A-1" -- and
# reporting that 1 as an invented quantity is exactly the kind of false
# positive that gets a warning ignored.
stripped = _UNIT_TOKEN.sub(" ", _WITH_UNIT.sub(" ", sentence))
found |= _numbers_in(stripped)

return found


def ungrounded_quantities(answer: str, evidence: str, prompt: str = "") -> set[str]:
"""Physical quantities in ``answer`` that appear in neither source.

Args:
answer: The reply shown to the user.
evidence: Everything the tools returned during the run.
prompt: The user's own turn, so a value they supplied is not reported
as invented when the agent repeats it back.

Returns:
Normalised numeric literals with no source. Empty means every physics
number in the answer is traceable to a tool or to the question.
"""
return quantities_in(answer) - _numbers_in(evidence) - _numbers_in(prompt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Match evidence claims, not arbitrary numeric tokens.

A tool return such as Process pk 60 finished makes Use ecutwfc 60 Ry appear grounded. PKs, counts, and exit statuses can therefore bless fabricated parameters. Preserve parameter/unit context when extracting evidence, rather than subtracting every number in the tool text.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` at line 121, Update the grounding calculation
around quantities_in and _numbers_in so evidence matching preserves
parameter/unit context instead of treating every numeric token as valid
evidence. Exclude identifiers such as process PKs, counts, and exit statuses
from grounding, while retaining tool-returned values that are explicitly
associated with the relevant parameter and unit; keep prompt-number subtraction
behavior intact.



def tool_output_text(messages: list[Any]) -> str:
"""Everything the tools returned during a run, flattened for searching.

Takes pydantic-ai messages rather than a prepared string so a caller can
hand over ``result.all_messages()`` without knowing the shape of a tool
return -- which is a dict or a dataclass as often as text.
"""
from pydantic_ai.messages import ModelRequest, ToolReturnPart

chunks: list[str] = []
for message in messages:
if not isinstance(message, ModelRequest):
continue
for part in message.parts:
if isinstance(part, ToolReturnPart):
content = part.content
chunks.append(content if isinstance(content, str) else repr(content))
return "\n".join(chunks)
2 changes: 1 addition & 1 deletion src/aiida_agents/tools/execution/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

from __future__ import annotations

from aiida_agents.tools.execution.run_context import query_run_context
from aiida_agents.tools.run_context import query_run_context
from aiida_agents.tools.execution.introspection import describe_workflow, list_workflows
from aiida_agents.tools.execution.protocol import build_workflow_inputs
from aiida_agents.tools.execution.schemas import (
Expand Down
1 change: 1 addition & 0 deletions tests/agents/analysis/test_analysis.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
EXPECTED_TOOLS = {
"get_process_status",
"get_process_report",
"query_run_context",
"list_retrieved_files",
"get_retrieved_file",
"list_processes",
Expand Down
2 changes: 1 addition & 1 deletion tests/agents/execution/test_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from pydantic_ai.tools import DeferredToolRequests

from aiida_agents.agents.execution import get_agent
from aiida_agents.tools.execution.run_context import query_run_context
from aiida_agents.tools.run_context import query_run_context
from aiida_agents.tools.execution.introspection import describe_workflow, list_workflows
from aiida_agents.tools.execution.schemas import WorkflowSpec
from aiida_agents.tools.execution.spec_execution import execute_workflow_spec
Expand Down
9 changes: 9 additions & 0 deletions tests/cli/test_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,15 @@ def __init__(self, output: object) -> None:
def new_messages(self) -> list[object]:
return []

def all_messages(self) -> list[object]:
"""No tool calls -- the grounding check reads these on every reply.

Empty rather than absent: a fake that omits it would make the check
raise, and making the check tolerate a missing method would let it skip
silently in production too.
"""
return []


def test_ask_prints_the_answer(monkeypatch: pytest.MonkeyPatch) -> None:
"""A one-shot `ask` prints the agent's reply and exits cleanly."""
Expand Down
Loading
Loading