Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/aiida_agents/agents/analysis/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
query_run_context,
search_structures,
)
from aiida_agents.rag import search_aiida_docs
from aiida_agents.rag import search_aiida_code, search_aiida_docs

# Every read tool is exposed through RetryOnToolError (see get_agent), so a
# tool that raises -- e.g. on a hallucinated or wrong-type identifier -- comes
Expand All @@ -59,6 +59,7 @@
search_structures,
query_run_context,
search_aiida_docs,
search_aiida_code,
]

_SYSTEM_PROMPT = (
Expand Down
16 changes: 15 additions & 1 deletion src/aiida_agents/agents/analysis/prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,21 @@ CRITICAL TOOL SELECTION RULES:
link_label, {"field": "type", ...} by link_type (e.g. 'create', 'return', 'call_calc'). Only
valid with 'with_incoming'/'with_outgoing' — not with 'with_ancestors'/'with_descendants'.
5. AIIDA DOCUMENTATION:
- For any conceptual questions, code example requests, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.

5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.
Comment on lines +78 to +85

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make the tool-selection rules mutually exclusive.

“How-to guidance” includes code-writing requests. Rule 5 can therefore direct the agent to search_aiida_docs instead of the required search_aiida_code. Exclude requests for Python code from Rule 5.

Proposed fix
-   - For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
+   - For conceptual questions, non-code how-to guidance, or non-code queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.
5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.
- For conceptual questions, non-code how-to guidance, or non-code queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.
5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/agents/analysis/prompt.md` around lines 78 - 85, Update the
conceptual and how-to guidance rule in the prompt so it explicitly excludes
requests for Python snippets, scripts, or code implementation; route those
requests exclusively through search_aiida_code(task=...), while retaining
search_aiida_docs(query=...) for other conceptual, how-to, import, and syntax
questions.

- Build the snippet out of what came back. Every AiiDA name you import must appear in one of
those examples. Writing AiiDA code from memory is the single most common way this fails:
a method name that looks right and does not exist costs the user a traceback and their trust.
- If the examples do not cover what was asked, say exactly that and show the closest one
returned. A partial answer that is true beats a complete one that is invented.
- Say which page each part came from, and keep the snippet minimal — no imports the code does
not use, no configuration the user did not ask about.
6. WORKFLOW/CALCULATION SUBMISSION — NOT YOURS:
- You are read-only and have no tool that submits, runs, or executes anything. Submission is
the Execution agent's responsibility.
Expand Down
42 changes: 32 additions & 10 deletions src/aiida_agents/cli/output.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,15 +150,37 @@ def _warn_ungrounded(text: str, messages: list[ModelMessage], question: str) ->
``aiida_agents.grounding``) but a false positive must cost a line of output,
never a withheld answer.
"""
from aiida_agents.grounding import tool_output_text, ungrounded_quantities
from aiida_agents.grounding import (
syntax_errors,
tool_output_text,
ungrounded_quantities,
ungrounded_symbols,
)

invented = ungrounded_quantities(text, tool_output_text(messages), question)
if not invented:
return
evidence = tool_output_text(messages)

values = ", ".join(sorted(invented))
console.print(
f"[yellow]⚠ Not found in any tool output: {values}. "
"Verify before using these values.[/yellow]"
)
logger.warning("ungrounded quantities in reply: %s", values)
invented = ungrounded_quantities(text, evidence, question)
if invented:
values = ", ".join(sorted(invented))
console.print(
f"[yellow]⚠ Not found in any tool output: {values}. "
"Verify before using these values.[/yellow]"
)
logger.warning("ungrounded quantities in reply: %s", values)

# The same question asked of generated code. Separate messages because the
# actions differ: an unsourced number wants checking, a name that exists in
# no example wants not being run.
unknown = ungrounded_symbols(text, evidence)
if unknown:
names = ", ".join(sorted(unknown))
console.print(
f"[yellow]⚠ Imported from AiiDA but in no documentation this run "
f"retrieved: {names}. Check these exist before running the "
"code.[/yellow]"
)
logger.warning("ungrounded symbols in reply: %s", names)

for problem in syntax_errors(text):
console.print(f"[yellow]⚠ The Python above does not parse: {problem}[/yellow]")
logger.warning("syntax error in generated code: %s", problem)
98 changes: 98 additions & 0 deletions src/aiida_agents/grounding.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,101 @@ def tool_output_text(messages: list[Any]) -> str:
content = part.content
chunks.append(content if isinstance(content, str) else repr(content))
return "\n".join(chunks)


# ---------------------------------------------------------------------------
# Code
#
# The quantity checks above ask whether a number in an answer came from a tool.
# A generated snippet raises the same question about names: an AiiDA class or
# function that exists in no retrieved example is the code equivalent of an
# invented cutoff, and it is worse, because the user finds out by running it.
#
# Two checks, deliberately narrow. Anything broader produces false positives,
# and this module's whole premise is that a warning nobody believes protects
# nothing -- a check that fires on `qb.append` because lists also have
# `append` would train people to ignore the one that matters.
# ---------------------------------------------------------------------------

#: A fenced block the answer presents as Python.
_PY_BLOCK = re.compile(
r"```(?:python|py|pycon)?\s*\n(?P<code>.*?)```", re.DOTALL | re.IGNORECASE
)


def python_blocks(answer: str) -> list[str]:
"""Every fenced code block in ``answer`` that is offered as Python.

An unlabelled fence counts: models routinely omit the language, and a
snippet the user will paste into ``verdi shell`` is Python whether or not
it said so.
"""
return [m.group("code") for m in _PY_BLOCK.finditer(answer)]


def syntax_errors(answer: str) -> list[str]:
"""Blocks in ``answer`` that are not parseable Python, with the reason.

Cheap, deterministic, and zero false positives on real code: if CPython
cannot parse it, no user can run it. Worth checking separately from the
symbol check because it catches the failure that wastes the most of a
user's time for the least excuse.

Console transcripts are skipped --- a block of ``verdi`` commands or
``>>>`` prompts is not meant to parse as a module.
"""
import ast

problems = []
for block in python_blocks(answer):
stripped = block.strip()
if not stripped or stripped.startswith((">>>", "$", "verdi ")):
continue
Comment on lines +215 to +218

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Skip transcript markers on every line.

A pycon block can have a comment or heading before its first >>> prompt. This code then calls ast.parse and emits a false syntax warning. Detect transcript markers in all non-empty lines. Add a regression test for a commented pycon transcript.

Proposed fix
     for block in python_blocks(answer):
         stripped = block.strip()
-        if not stripped or stripped.startswith((">>>", "$", "verdi ")):
+        lines = [line.lstrip() for line in stripped.splitlines()]
+        if not stripped or any(
+            line.startswith((">>>", "$", "verdi ")) for line in lines
+        ):
             continue
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for block in python_blocks(answer):
stripped = block.strip()
if not stripped or stripped.startswith((">>>", "$", "verdi ")):
continue
for block in python_blocks(answer):
stripped = block.strip()
lines = [line.lstrip() for line in stripped.splitlines()]
if not stripped or any(
line.startswith((">>>", "$", "verdi ")) for line in lines
):
continue
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` around lines 215 - 218, Update the block
filtering around python_blocks(answer) to inspect every non-empty line for
transcript markers, rather than checking only the first stripped line; skip
blocks containing >>>, $, or verdi-prefixed lines before ast.parse. Add a
regression test covering a commented or headed pycon transcript with a later
prompt and verify it produces no false syntax warning.

try:
ast.parse(stripped)
except SyntaxError as exc:
problems.append(f"{exc.msg} (line {exc.lineno})")
return problems


def _imported_aiida_names(code: str) -> set[str]:
"""Names the snippet imports from AiiDA, e.g. ``QueryBuilder``.

Only imports, and only from an ``aiida`` module. This is the narrowest
check that still catches the failure worth catching: a hallucinated class
or helper is almost always *imported*, and an import naming something that
does not exist fails at the very first line. Attribute and method names are
deliberately not checked --- ``.append``, ``.get``, ``.all`` belong to
builtins as often as to AiiDA, and flagging them would drown the signal.
"""
import ast

names: set[str] = set()
try:
tree = ast.parse(code)
except SyntaxError:
return names # reported by syntax_errors, not here

for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
return names
Comment on lines +244 to +249

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Check direct AiiDA imports.

import aiida.orm and import aiida.invented create ast.Import nodes. The current check ignores both, so unsupported AiiDA module imports never produce an ungrounded-symbol warning. Collect AiiDA module paths from ast.Import nodes and add a direct-import regression test.

Proposed fix
     for node in ast.walk(tree):
         if isinstance(node, ast.ImportFrom):
             module = node.module or ""
             if module == "aiida" or module.startswith("aiida."):
                 names.update(alias.name for alias in node.names)
+        elif isinstance(node, ast.Import):
+            names.update(
+                alias.name
+                for alias in node.names
+                if alias.name == "aiida" or alias.name.startswith("aiida.")
+            )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
return names
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
elif isinstance(node, ast.Import):
names.update(
alias.name
for alias in node.names
if alias.name == "aiida" or alias.name.startswith("aiida.")
)
return names
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` around lines 244 - 249, Update the AST import
collection logic to also handle ast.Import nodes, adding each directly imported
AiiDA module path such as aiida.orm or aiida.invented when it equals aiida or
starts with aiida.; preserve the existing ast.ImportFrom handling and add a
regression test covering direct AiiDA imports and the resulting
ungrounded-symbol warning.



def ungrounded_symbols(answer: str, evidence: str) -> set[str]:
"""AiiDA names imported in ``answer``'s code that no tool output mentions.

Args:
answer: The reply shown to the user, code blocks included.
evidence: Everything the tools returned during the run.

Returns:
Imported AiiDA names absent from every tool return. Empty means each
one appeared in something the agent actually retrieved.
"""
imported: set[str] = set()
for block in python_blocks(answer):
imported |= _imported_aiida_names(block)
return {name for name in imported if name not in evidence}
78 changes: 78 additions & 0 deletions src/aiida_agents/rag/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
"index_docs",
"index_plugin_corpora",
"query_docs",
"search_aiida_code",
"search_aiida_docs",
]

Expand Down Expand Up @@ -146,3 +147,80 @@ def search_aiida_docs(query: str) -> str:
formatted.append(f"{header}\n{url}\n{text}" if url else f"{header}\n{text}")

return "\n\n---\n\n".join(formatted)


#: Fence languages that mark a block as Python. The corpus is rendered by
#: ``rag._textbuild``, which carries each literal block's sphinx language
#: through into the fence, so this is a reliable marker rather than a guess.
_PYTHON_FENCES = ("```python", "```py\n", "```pycon", "```default")

#: How many hits to pull before filtering down to the ones carrying code. The
#: corpus is mostly prose, so the closest three chunks to "how do I query"
#: are often all explanation; casting wider and then filtering finds the
#: worked example that actually answers a code request.
_CODE_SEARCH_POOL = 12
_CODE_SEARCH_LIMIT = 4


def _has_python(text: str) -> bool:
return any(fence in text for fence in _PYTHON_FENCES)


def search_aiida_code(task: str) -> str:
"""Find real AiiDA code examples for a task, to write code from.

Use this **before writing any Python that uses AiiDA**, and prefer it over
``search_aiida_docs`` when the user wants a snippet, a script, or "how do I
do X in code". It returns worked examples from the official documentation
at the pinned version, so the APIs in them exist and have the signatures
shown.

Write your code from what this returns. Do **not** write AiiDA code from
memory: a plausible-looking method name that does not exist costs the user
a traceback and their trust, and it is the single most common way this
fails. If nothing here covers what you were asked, say so and show the
closest thing it did return rather than inventing the rest.

Args:
task: What the code needs to do, in natural language --- e.g.
"query for finished PwCalculations ordered by creation time".

Returns:
Up to four documentation excerpts that contain Python, each with its
source, section and a link to the page it came from.
"""
results = [
r
for r in query_docs(task, limit=_CODE_SEARCH_POOL)
if _has_python(r.get("text", ""))
]
if not results:
if not docs_index_available():
return _unavailable_message()
return (
"No AiiDA code examples found for this task -- the documentation "
"was searched, but nothing matching it contains a code example. "
"Say so, and do NOT write the code from memory: an invented API "
"is worse for the user than telling them the docs do not cover "
"this. Suggest `search_aiida_docs` for the concept, or point them "
"at the API reference."
)

formatted = []
for r in results[:_CODE_SEARCH_LIMIT]:
source = r.get("source", "unknown")
section = r.get("section", "")
corpus = r.get("corpus", "aiida-core")
location = f"{source} § {section}" if section else source
header = (
f"[{location}]" if corpus == "aiida-core" else f"[{corpus}: {location}]"
)
url = r.get("url", "")
body = (
f"{header}\n{url}\n{r.get('text', '')}"
if url
else (f"{header}\n{r.get('text', '')}")
)
formatted.append(body)

return "\n\n---\n\n".join(formatted)
1 change: 1 addition & 0 deletions tests/agents/analysis/test_analysis.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
"get_node_outputs",
"search_structures",
"search_aiida_docs",
"search_aiida_code",
}


Expand Down
75 changes: 75 additions & 0 deletions tests/rag/test_search_tool.py
Original file line number Diff line number Diff line change
Expand Up @@ -264,3 +264,78 @@ def test_a_result_predating_the_url_key_still_formats(self) -> None:
result = search_aiida_docs("q")

assert "Text A." in result


class TestSearchAiidaCode:
"""Retrieval aimed at writing code rather than explaining a concept."""

@staticmethod
def _hit(text: str, source: str = "howto/query") -> dict[str, str]:
return {"source": source, "section": "S", "text": text, "url": "u"}

def test_only_excerpts_containing_python_come_back(self) -> None:
"""The corpus is mostly prose; a code request wants the worked example."""
from aiida_agents.rag import search_aiida_code

fake = [
self._hit("Some prose about querying, with no example.", "topics/a"),
self._hit("Do this:\n```python\nqb = QueryBuilder()\n```", "howto/b"),
]
with patch("aiida_agents.rag.query_docs", return_value=fake):
result = search_aiida_code("query for calculations")

assert "qb = QueryBuilder()" in result
assert "topics/a" not in result

def test_it_casts_wider_than_the_prose_search(self) -> None:
"""Filtering to code-bearing hits only works with a pool to filter."""
from aiida_agents.rag import search_aiida_code

with (
patch("aiida_agents.rag.query_docs", return_value=[]) as mock_qd,
patch("aiida_agents.rag.docs_index_available", return_value=True),
):
search_aiida_code("anything")

assert mock_qd.call_args.kwargs["limit"] > 3

def test_no_code_found_forbids_answering_from_memory(self) -> None:
"""The whole failure mode is a plausible invented API."""
from aiida_agents.rag import search_aiida_code

fake = [self._hit("Prose only, no example here at all.")]
with (
patch("aiida_agents.rag.query_docs", return_value=fake),
patch("aiida_agents.rag.docs_index_available", return_value=True),
):
result = search_aiida_code("something obscure")

assert "do NOT write the code from memory" in result

def test_an_unbuilt_index_says_so_rather_than_reporting_no_examples(self) -> None:
from aiida_agents.rag import search_aiida_code

with (
patch("aiida_agents.rag.query_docs", return_value=[]),
patch("aiida_agents.rag.docs_index_available", return_value=False),
):
result = search_aiida_code("query")

assert "has not been built" in result

def test_the_page_link_travels_with_the_snippet(self) -> None:
"""So a user can read the example in its original context."""
from aiida_agents.rag import search_aiida_code

fake = [
{
"source": "howto/query",
"section": "S",
"text": "```python\nx = 1\n```",
"url": "https://example.invalid/howto/query.html#s",
}
]
with patch("aiida_agents.rag.query_docs", return_value=fake):
result = search_aiida_code("query")

assert "https://example.invalid/howto/query.html#s" in result
Loading
Loading