Skip to content

✨ agents: write AiiDA code from retrieved examples, and check it - #63

Merged
Jaweria-B merged 1 commit into
mainfrom
feature/write-aiida-code
Aug 1, 2026
Merged

Jaweria-B merged 1 commit into
mainfrom
feature/write-aiida-code

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Aug 1, 2026 •

Copy link
Copy Markdown
Collaborator

Writing code

The agents could explain how AiiDA works but never hand over a snippet to paste into verdi shell, which is what "how do I do X" usually means.

No new corpus was needed. The docs already carry 228 Python examples, and rag/_textbuild.py already preserves each one's fence and language — the code was in the index the whole time, and nothing was reaching for it.

search_aiida_code casts wider than the prose search (12 hits) and keeps only those containing Python: the closest three chunks to "how do I query" are usually all explanation, and the worked example sits further down.

Checking it

Saying "don't invent APIs" in a prompt is not enough, and this project has a measured result on that — an instruction not to invent numbers was ignored in five test runs out of five. So two checks read the answer afterwards, in the spirit of the quantity check that already runs on every reply.

syntax_errors parses each block with ast. Deterministic and false-positive-free: if CPython cannot parse it, nobody can run it. Shell and REPL transcripts are skipped — they are not meant to parse as a module.

ungrounded_symbols reports AiiDA names the snippet imports that appear in no tool output.

Imports only, deliberately. A hallucinated class is almost always imported and fails on the first line, whereas .append and .get belong to builtins as often as to AiiDA. A check that fired on qb.append would train people to ignore the one that matters, which is the premise the whole grounding module rests on.

Where the tool lives

On the Analysis agent, not a new specialist. Writing code changes nothing, and Analysis is the agent permitted to do anything that changes nothing. A third specialist would cost a routing decision on every request in the system and buy nothing until execution exists to justify it.

Second commit: a bug live testing found

Running the merged main against a real profile turned up a gap in the doc-links feature from #61. Asked "what is a CalcJobNode?", search_aiida_docs returned excerpts carrying correctly pinned readthedocs links — and the reply shown to the user contained no URL at all. The citation was in the tool output and never reached the prose.

Asking harder in the prompt is the wrong fix, for the reason above. Retrieval knows exactly which pages it returned; that is not a judgement call. So the CLI now prints them under the reply:

Sources:
  https://aiida-core.readthedocs.io/en/v2.8.0/topics/calculations/concepts.html#calcjobs

An inline citation, when the model does write one, is a bonus on top rather than the only path.

Testing

789 pass (+31), mypy clean across 132 files, ruff and mdformat green.

The pinned tool-surface test failed first, as designed — adding a tool has to be a deliberate edit in two places.

Not yet exercised against a real model: the free-tier quota on the test profile was exhausted partway through the manual run that found the citation bug.

The agents could explain how AiiDA works but never hand over a snippet
to paste into `verdi shell`, which is what users asking "how do I do X"
usually want. This adds that, built on the corpus already indexed -- the
docs carry 228 Python examples, and `_textbuild` already preserves each
one's fence and language, so no new index was needed to reach them.

`search_aiida_code` casts wider than the prose search and keeps only the
hits that actually contain Python: the closest three chunks to "how do I
query" are usually all explanation, and the worked example sits further
down. Its docstring and the prompt both say the same thing, because the
failure mode here is specific -- a method name that looks right and does
not exist costs the user a traceback and their trust.

Saying that in a prompt is not enough on its own, so two checks read the
answer afterwards, in the spirit of the quantity check that already runs
on every reply:

`syntax_errors` parses each block with ast. Deterministic, no false
positives -- if CPython cannot parse it, nobody can run it -- and it
catches the failure that wastes the most of a user's time. Shell and REPL
transcripts are skipped; they are not meant to parse as a module.

`ungrounded_symbols` reports AiiDA names the snippet imports that appear
in no tool output. Imports only, deliberately: a hallucinated class is
almost always imported and fails on the first line, while `.append` and
`.get` belong to builtins as often as to AiiDA. A check that fired on
those would train people to ignore the one that matters, which is this
module's whole premise.

The tool goes on the Analysis agent rather than a new specialist. Writing
code changes nothing, and Analysis is the agent allowed to do anything
that changes nothing; a third specialist would cost a routing decision on
every request and buy nothing until execution exists.
@coderabbitai

coderabbitai Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The analysis agent now searches documentation for Python examples before code generation. Grounding checks detect syntax errors and unsupported AiiDA imports, while CLI output reports each validation issue independently.

Changes

Code grounding

Layer / File(s) Summary
Python code search
src/aiida_agents/rag/__init__.py, tests/rag/test_search_tool.py
Adds search_aiida_code, which filters documentation results for Python examples, preserves source metadata, and handles unavailable or empty indexes.
Python grounding checks
src/aiida_agents/grounding.py, tests/test_grounding.py
Adds fenced-code extraction, syntax validation, and detection of ungrounded AiiDA imports.
Analysis validation and tool wiring
src/aiida_agents/agents/analysis/__init__.py, src/aiida_agents/agents/analysis/prompt.md, src/aiida_agents/cli/output.py, tests/agents/analysis/test_analysis.py
Registers the code-search tool, updates code-generation guidance, and reports quantity, symbol, and syntax validation issues separately.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AnalysisAgent
  participant search_aiida_code
  participant DocumentationIndex
  participant CLIOutput
  AnalysisAgent->>search_aiida_code: request Python examples
  search_aiida_code->>DocumentationIndex: retrieve code-bearing excerpts
  DocumentationIndex-->>search_aiida_code: return examples and URLs
  search_aiida_code-->>AnalysisAgent: provide grounded tool evidence
  AnalysisAgent->>CLIOutput: submit generated response
  CLIOutput-->>AnalysisAgent: report syntax and grounding warnings
Loading

Possibly related PRs

Suggested reviewers: geigerj2

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 46.15% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main changes: validating generated Python and retrieving grounded code examples.
Description check ✅ Passed The description directly explains the validation, retrieval, agent integration, CLI warnings, and tests in the changeset.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiida_agents/agents/analysis/prompt.md`:
- Around line 78-85: Update the conceptual and how-to guidance rule in the
prompt so it explicitly excludes requests for Python snippets, scripts, or code
implementation; route those requests exclusively through
search_aiida_code(task=...), while retaining search_aiida_docs(query=...) for
other conceptual, how-to, import, and syntax questions.

In `@src/aiida_agents/grounding.py`:
- Around line 244-249: Update the AST import collection logic to also handle
ast.Import nodes, adding each directly imported AiiDA module path such as
aiida.orm or aiida.invented when it equals aiida or starts with aiida.; preserve
the existing ast.ImportFrom handling and add a regression test covering direct
AiiDA imports and the resulting ungrounded-symbol warning.
- Around line 215-218: Update the block filtering around python_blocks(answer)
to inspect every non-empty line for transcript markers, rather than checking
only the first stripped line; skip blocks containing >>>, $, or verdi-prefixed
lines before ast.parse. Add a regression test covering a commented or headed
pycon transcript with a later prompt and verify it produces no false syntax
warning.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0f656030-985c-4fd8-87ec-75080e2a2dca

📥 Commits

Reviewing files that changed from the base of the PR and between 1a7e4f8 and 63d1e70.

📒 Files selected for processing (8)
  • src/aiida_agents/agents/analysis/__init__.py
  • src/aiida_agents/agents/analysis/prompt.md
  • src/aiida_agents/cli/output.py
  • src/aiida_agents/grounding.py
  • src/aiida_agents/rag/__init__.py
  • tests/agents/analysis/test_analysis.py
  • tests/rag/test_search_tool.py
  • tests/test_grounding.py

Comment on lines +78 to +85
- For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.

5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make the tool-selection rules mutually exclusive.

“How-to guidance” includes code-writing requests. Rule 5 can therefore direct the agent to search_aiida_docs instead of the required search_aiida_code. Exclude requests for Python code from Rule 5.

Proposed fix
-   - For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
+   - For conceptual questions, non-code how-to guidance, or non-code queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- For any conceptual questions, how-to guidance, or queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.
5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.
- For conceptual questions, non-code how-to guidance, or non-code queries about imports and syntax, you MUST call 'search_aiida_docs(query=...)' first instead of answering from memory.
- Cite the URL each excerpt carries, as a Markdown link on the thing it supports. Finding the
right page is the hardest part of these docs for most users.
5b. WRITING AIIDA CODE:
- If the user wants a snippet, a script, or asks "how do I do X in code", call
'search_aiida_code(task=...)' BEFORE writing any Python. It returns worked examples from the
documentation at the pinned version, so the APIs in them exist with the signatures shown.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/agents/analysis/prompt.md` around lines 78 - 85, Update the
conceptual and how-to guidance rule in the prompt so it explicitly excludes
requests for Python snippets, scripts, or code implementation; route those
requests exclusively through search_aiida_code(task=...), while retaining
search_aiida_docs(query=...) for other conceptual, how-to, import, and syntax
questions.

Comment on lines +215 to +218
for block in python_blocks(answer):
stripped = block.strip()
if not stripped or stripped.startswith((">>>", "$", "verdi ")):
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Skip transcript markers on every line.

A pycon block can have a comment or heading before its first >>> prompt. This code then calls ast.parse and emits a false syntax warning. Detect transcript markers in all non-empty lines. Add a regression test for a commented pycon transcript.

Proposed fix
     for block in python_blocks(answer):
         stripped = block.strip()
-        if not stripped or stripped.startswith((">>>", "$", "verdi ")):
+        lines = [line.lstrip() for line in stripped.splitlines()]
+        if not stripped or any(
+            line.startswith((">>>", "$", "verdi ")) for line in lines
+        ):
             continue
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for block in python_blocks(answer):
stripped = block.strip()
if not stripped or stripped.startswith((">>>", "$", "verdi ")):
continue
for block in python_blocks(answer):
stripped = block.strip()
lines = [line.lstrip() for line in stripped.splitlines()]
if not stripped or any(
line.startswith((">>>", "$", "verdi ")) for line in lines
):
continue
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` around lines 215 - 218, Update the block
filtering around python_blocks(answer) to inspect every non-empty line for
transcript markers, rather than checking only the first stripped line; skip
blocks containing >>>, $, or verdi-prefixed lines before ast.parse. Add a
regression test covering a commented or headed pycon transcript with a later
prompt and verify it produces no false syntax warning.

Comment on lines +244 to +249
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
return names

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Check direct AiiDA imports.

import aiida.orm and import aiida.invented create ast.Import nodes. The current check ignores both, so unsupported AiiDA module imports never produce an ungrounded-symbol warning. Collect AiiDA module paths from ast.Import nodes and add a direct-import regression test.

Proposed fix
     for node in ast.walk(tree):
         if isinstance(node, ast.ImportFrom):
             module = node.module or ""
             if module == "aiida" or module.startswith("aiida."):
                 names.update(alias.name for alias in node.names)
+        elif isinstance(node, ast.Import):
+            names.update(
+                alias.name
+                for alias in node.names
+                if alias.name == "aiida" or alias.name.startswith("aiida.")
+            )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
return names
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if module == "aiida" or module.startswith("aiida."):
names.update(alias.name for alias in node.names)
elif isinstance(node, ast.Import):
names.update(
alias.name
for alias in node.names
if alias.name == "aiida" or alias.name.startswith("aiida.")
)
return names
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` around lines 244 - 249, Update the AST import
collection logic to also handle ast.Import nodes, adding each directly imported
AiiDA module path such as aiida.orm or aiida.invented when it equals aiida or
starts with aiida.; preserve the existing ast.ImportFrom handling and add a
regression test covering direct AiiDA imports and the resulting
ungrounded-symbol warning.

@Jaweria-B
Jaweria-B merged commit 56799a0 into main Aug 1, 2026
5 checks passed
@Jaweria-B
Jaweria-B deleted the feature/write-aiida-code branch August 1, 2026 12:38
@Jaweria-B Jaweria-B changed the title Add code validation and retrieval for generated Python snippets ✨ agents: write AiiDA code from retrieved examples, and check it Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant