Skip to content

🐛 three faults found by testing the agents against a real profile - #68

Merged
Jaweria-B merged 1 commit into
mainfrom
fix/graceful-unknown-entity-and-noise
Aug 2, 2026
Merged

Jaweria-B merged 1 commit into
mainfrom
fix/graceful-unknown-entity-and-noise

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Aug 2, 2026 •

Copy link
Copy Markdown
Collaborator

All three came out of one session running real queries against a real AiiDA profile. None would have been caught by the unit suite.

1. query_nodes killed an entire run

Asked for PwCalculation on a profile where aiida-quantumespresso was not registered, the tool answered:

Unknown entity_type 'PwCalculation'. Did you mean: calcjobnode?

difflib rates those as close. So the model tried calcjobnode, failed differently, rephrased, and exhausted its retry budget — Tool 'query_nodes' exceeded max retries count of 3. The suggestion caused the loop.

Retrying is correct advice for a typo and wrong for a class belonging to a plugin nobody installed, so the two are now separated before any near match is offered. A CamelCase name ending in Calculation/WorkChain/CalcJob is told the plugin appears to be missing, told explicitly not to retry with a neighbouring name, and offered the generic types (calcjob, workchain, process, data) instead. An ordinary typo still gets its "did you mean".

2. The syntax check cried wolf

⚠ The Python above does not parse fired on any unlabelled fenced block — shell commands, YAML, pseudocode — because unlabelled fences were treated as Python.

That is precisely the failure grounding.py's own docstring warns about: a warning that fires on correct output teaches people to ignore it, and then it is not protecting the case it was built for.

Only fences explicitly labelled python are syntax-checked now. Symbol grounding still reads unlabelled blocks, where a non-Python block parses to nothing and cannot produce a false positive — so nothing is lost there.

Losing the check on unlabelled blocks is the cheaper mistake: the Codegen agent's prompt tells it to fence snippets as python, so the blocks that matter carry the label.

3. The planner mis-routed a request for code

"Give me a QueryBuilder snippet for finished PwCalculations" went to analysis, not codegen. The prompt said codegen was for "when the user asks for code", which was evidently not explicit enough. It now names the phrasings — "give me a snippet", "write me a query", "show me the Python", "a script that…" — and says plainly that asking to be handed code is always codegen, even where analysis could approximate an answer.

Prompt-level, so it needs a live re-test to confirm.

Testing

938 pass (+11), mypy clean across 148 files, ruff and mdformat green.

The tests for (1) use invented plugin-shaped names (FictionalCodeCalculation) rather than real ones like PwCalculation. Whether QE is installed varies with uv sync --group qe, and a test that passes only where a plugin is absent reports the environment rather than the code.

Also observed in that session, not fixed here

The sandbox's read-only containment could not be verified: sandbox init/check are PostgreSQL-only and the test profile uses SQLite. The rest of the sandbox path did verify end to end — a real QueryBuilder query returned 29,842 CalcJobNodes through it.

All three came from one session of running real queries, and none would
have been found by the unit suite.

query_nodes killed a whole run. Asked for PwCalculation on a profile
where aiida-quantumespresso was not registered, it answered "Unknown
entity_type ... Did you mean: calcjobnode?" -- difflib rates those as
close -- so the model tried that, failed differently, rephrased, and
exhausted its retry budget on "exceeded max retries count of 3".

Retrying is right for a typo and wrong for a class belonging to a plugin
nobody installed, so the two are now told apart before any suggestion is
offered. A CamelCase name ending in Calculation/WorkChain/CalcJob gets
told the plugin appears to be missing, told explicitly not to retry with
a neighbouring name, and offered the generic types instead. An ordinary
typo still gets its "did you mean".

The syntax check cried wolf. It flagged "the Python above does not
parse" on any unlabelled fenced block -- shell commands, YAML,
pseudocode -- because unlabelled fences were treated as Python. That is
precisely the failure this module's own docstring warns about: a warning
that fires on correct output teaches people to ignore it, and then it is
not protecting the case it exists for. Only fences explicitly labelled
python are syntax-checked now. Symbol grounding still reads unlabelled
blocks, where a non-Python block parses to nothing and cannot produce a
false positive.

The planner sent "give me a QueryBuilder snippet" to analysis rather
than codegen. The prompt said codegen was for "when the user asks for
code", which was evidently not explicit enough; it now names the
phrasings and says plainly that asking to be handed code is always
codegen, even where analysis could approximate an answer.

Tests for the first use invented plugin-shaped names rather than real
ones. Whether QE is installed varies with uv sync --group qe, and a test
that only passes where a plugin is absent reports the environment rather
than the code.
@coderabbitai

coderabbitai Bot commented Aug 2, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The planner now routes explicit code requests to codegen. Grounding validates only labelled Python fences. Query validation distinguishes unavailable plugin classes from ordinary unknown or misspelled entities.

Changes

Routing and validation refinements

Layer / File(s) Summary
Explicit code-request routing
src/aiida_agents/agents/planner/prompt.md
The planner routes code snippets, queries, Python, and scripts to codegen. Multi-condition routing remains separate.
Label-aware Python block validation
src/aiida_agents/grounding.py, tests/test_grounding.py
labelled_python_blocks filters explicitly labelled Python fences. syntax_errors ignores unlabelled and non-Python fences while symbol extraction still inspects unlabelled blocks.
Plugin-aware entity diagnostics
src/aiida_agents/tools/analysis/query_builder.py, tests/tools/analysis/test_query_builder.py
Unknown plugin-like names produce missing-plugin guidance without misleading suggestions. Ordinary misspellings retain near-match suggestions.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 76.47% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the three faults addressed by the pull request and identifies their discovery through real-profile agent testing.
Description check ✅ Passed The description directly explains all three changes, their causes, testing results, and intended behavior.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jaweria-B Jaweria-B changed the title Distinguish plugin classes from typos in entity validation 🐛 three faults found by testing the agents against a real profile Aug 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiida_agents/agents/planner/prompt.md`:
- Around line 55-61: Clarify the routing rule in the planner prompt so code-only
requests use codegen, while requests that also ask to prepare, submit, launch,
or preview a calculation use execution. Update the conflicting “always” wording
near the codegen guidance and add regression coverage for mixed
code-and-execution requests.

In `@src/aiida_agents/grounding.py`:
- Around line 187-189: Update the fenced-block parsing around `_PY_BLOCK` and
`python_blocks` to match complete fenced blocks regardless of language, then
filter labels using the existing Python-label behavior so non-Python blocks
cannot consume adjacent Python blocks. Update `labelled_python_blocks`
consistently, and add a regression test covering a non-Python block followed by
malformed labelled Python code reaching the validation paths.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6d190544-84e9-41cd-9f4e-77022494cf86

📥 Commits

Reviewing files that changed from the base of the PR and between aa9c053 and 75908c7.

📒 Files selected for processing (5)
  • src/aiida_agents/agents/planner/prompt.md
  • src/aiida_agents/grounding.py
  • src/aiida_agents/tools/analysis/query_builder.py
  • tests/test_grounding.py
  • tests/tools/analysis/test_query_builder.py

Comment on lines +55 to +61
**Asking for code means `codegen`, always.** "Give me a snippet", "write me a
query", "show me the Python", "a QueryBuilder for ...", "a script that ..." --
route these to `codegen` even when `analysis` could approximate the answer with
a tool. The user asked to be handed code they can run and adapt; a prose
summary of their data is not that, and `analysis` cannot produce one.

Choose `codegen` also when the question combines

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Clarify precedence between code requests and execution requests.

The new “always” rule conflicts with Lines 50-53, which route setup, submission, preparation, and launch requests to execution. A request such as “write a Python script and submit it” matches both rules, but codegen cannot submit or change anything.

Route code-only requests to codegen. Retain execution when the request also asks to prepare, submit, launch, or preview a calculation. Add regression tests for mixed requests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/agents/planner/prompt.md` around lines 55 - 61, Clarify the
routing rule in the planner prompt so code-only requests use codegen, while
requests that also ask to prepare, submit, launch, or preview a calculation use
execution. Update the conflicting “always” wording near the codegen guidance and
add regression coverage for mixed code-and-execution requests.

Comment on lines 187 to +189
_PY_BLOCK = re.compile(
r"```(?:python|py|pycon)?\s*\n(?P<code>.*?)```", re.DOTALL | re.IGNORECASE
r"```(?P<lang>python|py|pycon)?[ \t]*\n(?P<code>.*?)```",
re.DOTALL | re.IGNORECASE,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Parse all fenced blocks before filtering by language.

_PY_BLOCK can start at the closing fence of a non-Python block. If a bash block precedes a python block, the matcher can consume from the bash closing fence through the python opening fence. The labelled Python block then does not reach syntax_errors or ungrounded_symbols.

Match complete fenced blocks regardless of label. Filter the parsed blocks afterward. Add a regression test with a non-Python block followed by malformed labelled Python.

Proposed fix
-_PY_BLOCK = re.compile(
-    r"```(?P<lang>python|py|pycon)?[ \t]*\n(?P<code>.*?)```",
-    re.DOTALL | re.IGNORECASE,
+_FENCED_BLOCK = re.compile(
+    r"^```(?P<lang>[^\n`]*)\n(?P<code>.*?)^```[ \t]*$",
+    re.DOTALL | re.MULTILINE,
 )
+_PYTHON_LABELS = {"", "python", "py", "pycon"}
 
 def python_blocks(answer: str) -> list[str]:
-    return [m.group("code") for m in _PY_BLOCK.finditer(answer)]
+    return [
+        m.group("code")
+        for m in _FENCED_BLOCK.finditer(answer)
+        if m.group("lang").strip().casefold() in _PYTHON_LABELS
+    ]
 
 def labelled_python_blocks(answer: str) -> list[str]:
-    return [m.group("code") for m in _PY_BLOCK.finditer(answer) if m.group("lang")]
+    return [
+        m.group("code")
+        for m in _FENCED_BLOCK.finditer(answer)
+        if m.group("lang").strip().casefold() in _PYTHON_LABELS - {""}
+    ]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/grounding.py` around lines 187 - 189, Update the
fenced-block parsing around `_PY_BLOCK` and `python_blocks` to match complete
fenced blocks regardless of language, then filter labels using the existing
Python-label behavior so non-Python blocks cannot consume adjacent Python
blocks. Update `labelled_python_blocks` consistently, and add a regression test
covering a non-Python block followed by malformed labelled Python code reaching
the validation paths.

@Jaweria-B
Jaweria-B merged commit 611d0e2 into main Aug 2, 2026
5 checks passed
@Jaweria-B
Jaweria-B deleted the fix/graceful-unknown-entity-and-noise branch August 2, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant