Repository navigation
Add cutoff range checking against pseudopotential recommendations - #54
Conversation
… expect ADR-07 built the schema tier and deferred this one. Since then every write has been guarded by code except the question that decides whether the calculation is worth running at all: a cutoff of 12 Ry passes schema validation, passes `spec.inputs.validate()`, submits cleanly, and returns a converged, authoritative, wrong number. The only thing between that and an HPC queue was a human reading a diff and knowing the right value by heart. There is an authority for that value and it is not this package. A pseudopotential family ships the cutoffs its authors converged it against, and aiida-pseudo exposes them per element, so `check_input_ranges` compares against what the family recommends for *this structure's elements* and reports the difference. It carries no table of its own; where no family can be identified it says nothing rather than inventing a number. It warns, it does not block. A low cutoff is a legitimate smoke test and a high one a converged production run, so refusing the submission would be wrong about as often as it was right. What was missing is the fact reaching the person approving, at the moment they approve -- so the finding is printed at the approval prompt, and offered to the agent as a tool. Same belt-and-braces as the grounding check: the prompt asks the model to check, and the code checks regardless, because an instruction can be ignored and this is the last moment before core-hours are spent. Only cutoffs. No installed package publishes a recommended k-point spacing, so a bound on `kpoints_distance` would be a number this file invented and then presented with a validator's authority -- the exact failure the project is built against. It is left alone, and the prompt says why, so an empty result is not read as a clean bill of health. The historical case is the one that most needs this: an `ecutwfc` taken from `query_run_context` came from runs that may have used a different family entirely, and a value that was right there can be well under-converged here. aiida-pseudo becomes a test dependency, not a runtime one. The source reads the family through AiiDA's group API and imports nothing from it; the tests build a real family and set real cutoffs, because mocking `get_recommended_cutoffs` would have removed the only part worth doubting -- that aiida-pseudo is asked correctly, for the right elements, in the right unit.
📝 WalkthroughWalkthroughThe change adds ChangesCutoff validation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ExecutionAgent
participant check_input_ranges
participant PseudopotentialFamily
participant ApprovalPreview
ExecutionAgent->>check_input_ranges: validate WorkflowSpec cutoffs
check_input_ranges->>PseudopotentialFamily: retrieve cutoff recommendations
PseudopotentialFamily-->>check_input_ranges: return recommendations
check_input_ranges-->>ExecutionAgent: return RangeFinding results
ExecutionAgent->>ApprovalPreview: render resolved inputs and warnings
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/aiida_agents/cli/hitl.py (1)
132-151: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider a direct test asserting the approval preview surfaces a range finding.
_range_warningsand its use in_print_previews(Line 173) are new, user-visible safety behavior for the approval flow.tests/tools/execution/test_ranges.pythoroughly coverscheck_input_rangesitself, but none of the files in this review exercise_print_previews/_range_warningsdirectly (e.g., asserting that a submission preview with an out-of-range cutoff prints the⚠️line). If such a test exists elsewhere it is out of scope here; otherwise, add one so a future refactor of_submission_argsor_print_previewscannot silently drop the warning without a test failing.Also applies to: 173-174
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/cli/hitl.py` around lines 132 - 151, Add a focused test covering the approval preview path through _print_previews and _range_warnings: build a submission with an out-of-range cutoff, invoke the preview, and assert the output includes the warning line marked with ⚠️. Reuse existing test helpers and preserve the current check_input_ranges behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/tools/execution/ranges.py`:
- Around line 121-143: Update _cutoffs_in to retain every valid cutoff
occurrence rather than discarding later values with the key-in-found guard; use
a collection that preserves each path and numeric value. Adjust
check_input_ranges to iterate through all returned occurrences and evaluate each
independently, while retaining deterministic ordering and existing invalid-value
handling.
---
Nitpick comments:
In `@src/aiida_agents/cli/hitl.py`:
- Around line 132-151: Add a focused test covering the approval preview path
through _print_previews and _range_warnings: build a submission with an
out-of-range cutoff, invoke the preview, and assert the output includes the
warning line marked with ⚠️. Reuse existing test helpers and preserve the
current check_input_ranges behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 36efe03c-37ed-4432-a1b2-55e2e4382713
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
pyproject.tomlsrc/aiida_agents/agents/execution/__init__.pysrc/aiida_agents/agents/execution/prompt.mdsrc/aiida_agents/cli/hitl.pysrc/aiida_agents/mcp/tools/__init__.pysrc/aiida_agents/tools/__init__.pysrc/aiida_agents/tools/execution/__init__.pysrc/aiida_agents/tools/execution/ranges.pytests/agents/execution/test_integration.pytests/tools/execution/test_ranges.py
| def _cutoffs_in(inputs: dict[str, t.Any]) -> dict[str, tuple[str, float]]: | ||
| """Cutoffs found anywhere in the spec, keyed by name. | ||
|
|
||
| Returns the first of each: a spec that sets ``ecutwfc`` in two namespaces | ||
| is describing two sub-calculations, and reporting the same finding twice | ||
| would clutter an approval prompt without adding anything. | ||
| """ | ||
| found: dict[str, tuple[str, float]] = {} | ||
| for params_path, params in _find_leaf(inputs, "parameters"): | ||
| if not isinstance(params, dict): | ||
| continue | ||
| for card in _SYSTEM_CARDS: | ||
| section = params.get(card) | ||
| if not isinstance(section, dict): | ||
| continue | ||
| for key in _CUTOFF_KEYS: | ||
| if key in found or key not in section: | ||
| continue | ||
| try: | ||
| found[key] = (f"{params_path}.{card}.{key}", float(section[key])) | ||
| except (TypeError, ValueError): | ||
| logger.debug("non-numeric %s at %s", key, params_path) | ||
| return found |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
_cutoffs_in checks only the first occurrence of each cutoff key, silently skipping a legitimately different value in a second namespace.
_cutoffs_in returns a dict keyed by cutoff name and keeps only the first (path, value) per key (Line 137: if key in found or key not in section: continue). The docstring justifies this by assuming a duplicate is "describing two sub-calculations" reporting "the same finding twice." That assumption does not hold in general: workflows such as PwRelaxWorkChain carry independent parameter namespaces (a relax namespace and a separate final-SCF-style namespace) that can legitimately use different cutoffs. If the first namespace's cutoff matches the recommendation but the second namespace's does not, check_input_ranges never reports it, because check_input_ranges (Line 291) iterates sorted(cutoffs.items()), which has already discarded every occurrence but the first.
This defeats the stated purpose of the check: catching "a cutoff of 12 Ry" that "passes schema validation ... and produces a converged, authoritative, wrong number" (Lines 5-7), specifically for the sub-calculation the tool never looks at.
Return every occurrence instead of the first, and check each one.
🐛 Proposed fix to check every cutoff occurrence, not just the first
-def _cutoffs_in(inputs: dict[str, t.Any]) -> dict[str, tuple[str, float]]:
- """Cutoffs found anywhere in the spec, keyed by name.
-
- Returns the first of each: a spec that sets ``ecutwfc`` in two namespaces
- is describing two sub-calculations, and reporting the same finding twice
- would clutter an approval prompt without adding anything.
- """
- found: dict[str, tuple[str, float]] = {}
+def _cutoffs_in(inputs: dict[str, t.Any]) -> list[tuple[str, str, float]]:
+ """Every cutoff found anywhere in the spec, as ``(key, path, value)`` triples.
+
+ All occurrences are returned, not just the first: a workflow such as
+ ``PwRelaxWorkChain`` carries independent namespaces (a relax stage and a
+ final-SCF stage) that can legitimately use different cutoffs, and a bad
+ value in the second one must still be checked.
+ """
+ found: list[tuple[str, str, float]] = []
for params_path, params in _find_leaf(inputs, "parameters"):
if not isinstance(params, dict):
continue
for card in _SYSTEM_CARDS:
section = params.get(card)
if not isinstance(section, dict):
continue
for key in _CUTOFF_KEYS:
- if key in found or key not in section:
+ if key not in section:
continue
try:
- found[key] = (f"{params_path}.{card}.{key}", float(section[key]))
+ found.append(
+ (key, f"{params_path}.{card}.{key}", float(section[key]))
+ )
except (TypeError, ValueError):
logger.debug("non-numeric %s at %s", key, params_path)
return foundAnd update the caller's loop accordingly:
- for key, (path, value) in sorted(cutoffs.items()):
+ for key, path, value in sorted(cutoffs):
expected = recommended.get(key)Also applies to: 290-322
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/tools/execution/ranges.py` around lines 121 - 143, Update
_cutoffs_in to retain every valid cutoff occurrence rather than discarding later
values with the key-in-found guard; use a collection that preserves each path
and numeric value. Adjust check_input_ranges to iterate through all returned
occurrences and evaluate each independently, while retaining deterministic
ordering and existing invalid-value handling.
The overview's "what is deliberately absent" section said physics-range validation was deferred and that batch writes had no path. Both shipped --- in #54 and #57 --- and neither line was updated. This page is the first thing a reader reaches for, so it was actively telling them two capabilities did not exist. Replaced with what is *actually* absent, which in both cases is narrower and more useful than "not built": - physics checking exists but does not block, because a low cutoff is under-converged for production and reasonable for a smoke test, so refusing would be wrong about as often as it was right; - k-point spacing is unchecked because no installed package publishes a recommended one, and inventing a bound is the failure this project is built against; - workflow *construction* is genuinely absent: sequences are composed by chaining existing workflows, and nothing emits a new WorkChain class. The Execution agent's description and the grounding check's both drifted the same way and are brought up to date with the tools they now hold.
* ✨ agents: make the handoff between specialists a typed message The proposal asked for an agent-to-agent communication protocol. What existed was a formatted string: the CLI pasted step one's prose answer into step two's prompt under a heading. It worked, and it lost the thing that mattered most. A diagnosis says "the failure is in PwCalculation pk 334407". The next step needs 334407, and with a text handoff the only route to it is a second model reading that sentence and pulling the number back out. That is a transcription step between two language models sitting directly in front of a write, and a resubmission aimed at a mistyped pk is the expensive kind of wrong. `Handoff` carries two halves: the producing agent's findings verbatim, and the node references its *tools* returned -- extracted from tool output, not from the answer, because a pk the agent wrote down may be a typo of one it was given and a pk taken from the tool return cannot be. The receiving agent is told to use those identifiers directly rather than re-read them out of the prose. Three things it deliberately is not. Not a transport. The protocol is satisfied as a defined message with a producer, a consumer and typed content, not as a channel between processes. Both specialists run in one interpreter against one profile, so a transport would add moving parts for no gain -- and routing a specialist's output anywhere but the CLI would break the approval guarantee ADR-09 is built around. Not a summary. The findings pass through unchanged; compressing them would put a third party in the position of rewriting a conclusion it did not reach. Not unbounded. References are capped at twenty, because a query matching hundreds would crowd out the findings they exist to support, and the next step can always query again. The REPL extracts from the turn's own messages rather than the accumulated history: handing a later step pks from an earlier, unrelated question and calling them "what this step found" would be false. ADR-09 and the architecture overview are revised to match -- both still described the handoff as labelled text. * 📚 architecture: stop claiming two shipped features are absent The overview's "what is deliberately absent" section said physics-range validation was deferred and that batch writes had no path. Both shipped --- in #54 and #57 --- and neither line was updated. This page is the first thing a reader reaches for, so it was actively telling them two capabilities did not exist. Replaced with what is *actually* absent, which in both cases is narrower and more useful than "not built": - physics checking exists but does not block, because a low cutoff is under-converged for production and reasonable for a smoke test, so refusing would be wrong about as often as it was right; - k-point spacing is unchecked because no installed package publishes a recommended one, and inventing a bound is the failure this project is built against; - workflow *construction* is genuinely absent: sequences are composed by chaining existing workflows, and nothing emits a new WorkChain class. The Execution agent's description and the grounding check's both drifted the same way and are brought up to date with the tools they now hold. * 📚 rewrite the README for a project that has features It still said "engineering scaffolding is in place; feature work has not started", which stopped being true around twenty tools ago. It is the first thing anyone evaluating this reads, and it was describing an empty repository. Now: what the thing does, how to install and run it, and what it cannot do. The capability section is grouped by what a user wants rather than by module, and each entry says the non-obvious part -- that a diagnosis reports which remedies the work chain *already tried*, that the approval guarantee lives on the tool rather than in a prompt, that every reply is checked for quantities no tool produced. Those are the reasons to use it over a chat window with the docs pasted in. The quickstart is verified rather than assumed: every command named exists in `--help`, and the provider/model example matches `.env.example` instead of inventing a model string. A "known limits" section, because the honest ones are short and a reader finding them out by surprise is worse than a reader who was told. No release. It composes workflows rather than writing them. Physics checks warn rather than block. Only Quantum ESPRESSO has been driven against real failures. The example output is marked as illustrative: its pks, exit code and handler name are from a real Quantum ESPRESSO diagnosis, but the prose is abridged and it is not a verbatim transcript.
Summary
This PR adds a new
check_input_ranges()tool that validates workflow input cutoffs against the recommendations published by their pseudopotential families. The check warns users when cutoffs deviate materially from what the family's authors converged them against, helping catch both under-converged calculations (which produce wrong results silently) and unnecessarily expensive over-converged ones.Key Changes
New module
ranges.py: Implementscheck_input_ranges()function that:ecutwfc,ecutrho) from workflow specs at any nesting levelRangeFindingobjects with severity, source attribution, and user-friendly messagesIntegration into approval workflow:
_range_warnings()helper inhitl.pythat runs the check on every submission previewAgent prompt updates: Added step 4c instructing the agent to call
check_input_ranges()whenever it sets or modifies cutoffsTool registration: Exported
check_input_rangesthrough the tools module hierarchy and MCP interfaceComprehensive test suite: 235 lines of tests covering:
Implementation Details
aiida-pseudofamilies publish, not against hardcoded values in this packagekpoints_distance(which has no equivalent published recommendation)aiida-pseudois added as a test dependency since the package itself remains code-agnostic and only reads families through AiiDA's standard group API