Repository navigation
🐛 three faults found by testing the agents against a real profile - #68
Conversation
All three came from one session of running real queries, and none would have been found by the unit suite. query_nodes killed a whole run. Asked for PwCalculation on a profile where aiida-quantumespresso was not registered, it answered "Unknown entity_type ... Did you mean: calcjobnode?" -- difflib rates those as close -- so the model tried that, failed differently, rephrased, and exhausted its retry budget on "exceeded max retries count of 3". Retrying is right for a typo and wrong for a class belonging to a plugin nobody installed, so the two are now told apart before any suggestion is offered. A CamelCase name ending in Calculation/WorkChain/CalcJob gets told the plugin appears to be missing, told explicitly not to retry with a neighbouring name, and offered the generic types instead. An ordinary typo still gets its "did you mean". The syntax check cried wolf. It flagged "the Python above does not parse" on any unlabelled fenced block -- shell commands, YAML, pseudocode -- because unlabelled fences were treated as Python. That is precisely the failure this module's own docstring warns about: a warning that fires on correct output teaches people to ignore it, and then it is not protecting the case it exists for. Only fences explicitly labelled python are syntax-checked now. Symbol grounding still reads unlabelled blocks, where a non-Python block parses to nothing and cannot produce a false positive. The planner sent "give me a QueryBuilder snippet" to analysis rather than codegen. The prompt said codegen was for "when the user asks for code", which was evidently not explicit enough; it now names the phrasings and says plainly that asking to be handed code is always codegen, even where analysis could approximate an answer. Tests for the first use invented plugin-shaped names rather than real ones. Whether QE is installed varies with uv sync --group qe, and a test that only passes where a plugin is absent reports the environment rather than the code.
📝 WalkthroughWalkthroughThe planner now routes explicit code requests to ChangesRouting and validation refinements
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/agents/planner/prompt.md`:
- Around line 55-61: Clarify the routing rule in the planner prompt so code-only
requests use codegen, while requests that also ask to prepare, submit, launch,
or preview a calculation use execution. Update the conflicting “always” wording
near the codegen guidance and add regression coverage for mixed
code-and-execution requests.
In `@src/aiida_agents/grounding.py`:
- Around line 187-189: Update the fenced-block parsing around `_PY_BLOCK` and
`python_blocks` to match complete fenced blocks regardless of language, then
filter labels using the existing Python-label behavior so non-Python blocks
cannot consume adjacent Python blocks. Update `labelled_python_blocks`
consistently, and add a regression test covering a non-Python block followed by
malformed labelled Python code reaching the validation paths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6d190544-84e9-41cd-9f4e-77022494cf86
📒 Files selected for processing (5)
src/aiida_agents/agents/planner/prompt.mdsrc/aiida_agents/grounding.pysrc/aiida_agents/tools/analysis/query_builder.pytests/test_grounding.pytests/tools/analysis/test_query_builder.py
| **Asking for code means `codegen`, always.** "Give me a snippet", "write me a | ||
| query", "show me the Python", "a QueryBuilder for ...", "a script that ..." -- | ||
| route these to `codegen` even when `analysis` could approximate the answer with | ||
| a tool. The user asked to be handed code they can run and adapt; a prose | ||
| summary of their data is not that, and `analysis` cannot produce one. | ||
|
|
||
| Choose `codegen` also when the question combines |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Clarify precedence between code requests and execution requests.
The new “always” rule conflicts with Lines 50-53, which route setup, submission, preparation, and launch requests to execution. A request such as “write a Python script and submit it” matches both rules, but codegen cannot submit or change anything.
Route code-only requests to codegen. Retain execution when the request also asks to prepare, submit, launch, or preview a calculation. Add regression tests for mixed requests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/agents/planner/prompt.md` around lines 55 - 61, Clarify the
routing rule in the planner prompt so code-only requests use codegen, while
requests that also ask to prepare, submit, launch, or preview a calculation use
execution. Update the conflicting “always” wording near the codegen guidance and
add regression coverage for mixed code-and-execution requests.
| _PY_BLOCK = re.compile( | ||
| r"```(?:python|py|pycon)?\s*\n(?P<code>.*?)```", re.DOTALL | re.IGNORECASE | ||
| r"```(?P<lang>python|py|pycon)?[ \t]*\n(?P<code>.*?)```", | ||
| re.DOTALL | re.IGNORECASE, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Parse all fenced blocks before filtering by language.
_PY_BLOCK can start at the closing fence of a non-Python block. If a bash block precedes a python block, the matcher can consume from the bash closing fence through the python opening fence. The labelled Python block then does not reach syntax_errors or ungrounded_symbols.
Match complete fenced blocks regardless of label. Filter the parsed blocks afterward. Add a regression test with a non-Python block followed by malformed labelled Python.
Proposed fix
-_PY_BLOCK = re.compile(
- r"```(?P<lang>python|py|pycon)?[ \t]*\n(?P<code>.*?)```",
- re.DOTALL | re.IGNORECASE,
+_FENCED_BLOCK = re.compile(
+ r"^```(?P<lang>[^\n`]*)\n(?P<code>.*?)^```[ \t]*$",
+ re.DOTALL | re.MULTILINE,
)
+_PYTHON_LABELS = {"", "python", "py", "pycon"}
def python_blocks(answer: str) -> list[str]:
- return [m.group("code") for m in _PY_BLOCK.finditer(answer)]
+ return [
+ m.group("code")
+ for m in _FENCED_BLOCK.finditer(answer)
+ if m.group("lang").strip().casefold() in _PYTHON_LABELS
+ ]
def labelled_python_blocks(answer: str) -> list[str]:
- return [m.group("code") for m in _PY_BLOCK.finditer(answer) if m.group("lang")]
+ return [
+ m.group("code")
+ for m in _FENCED_BLOCK.finditer(answer)
+ if m.group("lang").strip().casefold() in _PYTHON_LABELS - {""}
+ ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/grounding.py` around lines 187 - 189, Update the
fenced-block parsing around `_PY_BLOCK` and `python_blocks` to match complete
fenced blocks regardless of language, then filter labels using the existing
Python-label behavior so non-Python blocks cannot consume adjacent Python
blocks. Update `labelled_python_blocks` consistently, and add a regression test
covering a non-Python block followed by malformed labelled Python code reaching
the validation paths.
All three came out of one session running real queries against a real AiiDA profile. None would have been caught by the unit suite.
1.
query_nodeskilled an entire runAsked for
PwCalculationon a profile whereaiida-quantumespressowas not registered, the tool answered:difflibrates those as close. So the model triedcalcjobnode, failed differently, rephrased, and exhausted its retry budget —Tool 'query_nodes' exceeded max retries count of 3. The suggestion caused the loop.Retrying is correct advice for a typo and wrong for a class belonging to a plugin nobody installed, so the two are now separated before any near match is offered. A CamelCase name ending in
Calculation/WorkChain/CalcJobis told the plugin appears to be missing, told explicitly not to retry with a neighbouring name, and offered the generic types (calcjob,workchain,process,data) instead. An ordinary typo still gets its "did you mean".2. The syntax check cried wolf
⚠ The Python above does not parsefired on any unlabelled fenced block — shell commands, YAML, pseudocode — because unlabelled fences were treated as Python.That is precisely the failure
grounding.py's own docstring warns about: a warning that fires on correct output teaches people to ignore it, and then it is not protecting the case it was built for.Only fences explicitly labelled
pythonare syntax-checked now. Symbol grounding still reads unlabelled blocks, where a non-Python block parses to nothing and cannot produce a false positive — so nothing is lost there.Losing the check on unlabelled blocks is the cheaper mistake: the Codegen agent's prompt tells it to fence snippets as
python, so the blocks that matter carry the label.3. The planner mis-routed a request for code
"Give me a QueryBuilder snippet for finished PwCalculations" went to
analysis, notcodegen. The prompt said codegen was for "when the user asks for code", which was evidently not explicit enough. It now names the phrasings — "give me a snippet", "write me a query", "show me the Python", "a script that…" — and says plainly that asking to be handed code is always codegen, even where analysis could approximate an answer.Prompt-level, so it needs a live re-test to confirm.
Testing
938 pass (+11), mypy clean across 148 files, ruff and mdformat green.
The tests for (1) use invented plugin-shaped names (
FictionalCodeCalculation) rather than real ones likePwCalculation. Whether QE is installed varies withuv sync --group qe, and a test that passes only where a plugin is absent reports the environment rather than the code.Also observed in that session, not fixed here
The sandbox's read-only containment could not be verified:
sandbox init/checkare PostgreSQL-only and the test profile uses SQLite. The rest of the sandbox path did verify end to end — a real QueryBuilder query returned 29,842 CalcJobNodes through it.