Skip to content

Add cutoff range checking against pseudopotential recommendations - #54

Merged
Jaweria-B merged 1 commit into
mainfrom
feature/physics-range-validation
Jul 31, 2026
Merged

Jaweria-B merged 1 commit into
mainfrom
feature/physics-range-validation

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Jul 31, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds a new check_input_ranges() tool that validates workflow input cutoffs against the recommendations published by their pseudopotential families. The check warns users when cutoffs deviate materially from what the family's authors converged them against, helping catch both under-converged calculations (which produce wrong results silently) and unnecessarily expensive over-converged ones.

Key Changes

  • New module ranges.py: Implements check_input_ranges() function that:

    • Extracts cutoff parameters (ecutwfc, ecutrho) from workflow specs at any nesting level
    • Resolves the structure and pseudopotential family from the spec
    • Queries the family's published recommended cutoffs for the structure's elements
    • Reports findings when cutoffs fall below 90% of recommended (convergence risk) or exceed 3x recommended (cost risk)
    • Returns structured RangeFinding objects with severity, source attribution, and user-friendly messages
  • Integration into approval workflow:

    • Added _range_warnings() helper in hitl.py that runs the check on every submission preview
    • Displays cutoff warnings alongside other submission details in the approval prompt
    • Gracefully handles check failures (logs and continues rather than blocking approval)
  • Agent prompt updates: Added step 4c instructing the agent to call check_input_ranges() whenever it sets or modifies cutoffs

  • Tool registration: Exported check_input_ranges through the tools module hierarchy and MCP interface

  • Comprehensive test suite: 235 lines of tests covering:

    • Comparison logic against real pseudopotential families
    • Finding cutoffs across different spec nesting patterns
    • Both uppercase and lowercase card spellings
    • Family identification from labels and pseudo nodes
    • Graceful handling of missing structure/family/cutoffs
    • Malformed input resilience

Implementation Details

  • Authority-based validation: The check compares against what aiida-pseudo families publish, not against hardcoded values in this package
  • Warnings only: Deliberately does not block submissions—low cutoffs for smoke tests and high cutoffs for production runs are both legitimate
  • Selective scope: Only checks cutoffs (which have an external authority); deliberately ignores kpoints_distance (which has no equivalent published recommendation)
  • Robust path finding: Uses leaf-name matching rather than hardcoded paths, so it works across different workflow types and plugin layouts
  • Test dependency only: aiida-pseudo is added as a test dependency since the package itself remains code-agnostic and only reads families through AiiDA's standard group API

… expect

ADR-07 built the schema tier and deferred this one. Since then every write has
been guarded by code except the question that decides whether the calculation
is worth running at all: a cutoff of 12 Ry passes schema validation, passes
`spec.inputs.validate()`, submits cleanly, and returns a converged,
authoritative, wrong number. The only thing between that and an HPC queue was a
human reading a diff and knowing the right value by heart.

There is an authority for that value and it is not this package. A
pseudopotential family ships the cutoffs its authors converged it against, and
aiida-pseudo exposes them per element, so `check_input_ranges` compares against
what the family recommends for *this structure's elements* and reports the
difference. It carries no table of its own; where no family can be identified it
says nothing rather than inventing a number.

It warns, it does not block. A low cutoff is a legitimate smoke test and a high
one a converged production run, so refusing the submission would be wrong about
as often as it was right. What was missing is the fact reaching the person
approving, at the moment they approve -- so the finding is printed at the
approval prompt, and offered to the agent as a tool. Same belt-and-braces as the
grounding check: the prompt asks the model to check, and the code checks
regardless, because an instruction can be ignored and this is the last moment
before core-hours are spent.

Only cutoffs. No installed package publishes a recommended k-point spacing, so
a bound on `kpoints_distance` would be a number this file invented and then
presented with a validator's authority -- the exact failure the project is built
against. It is left alone, and the prompt says why, so an empty result is not
read as a clean bill of health.

The historical case is the one that most needs this: an `ecutwfc` taken from
`query_run_context` came from runs that may have used a different family
entirely, and a value that was right there can be well under-converged here.

aiida-pseudo becomes a test dependency, not a runtime one. The source reads the
family through AiiDA's group API and imports nothing from it; the tests build a
real family and set real cutoffs, because mocking `get_recommended_cutoffs`
would have removed the only part worth doubting -- that aiida-pseudo is asked
correctly, for the right elements, in the right unit.
@coderabbitai

coderabbitai Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The change adds check_input_ranges to compare workflow cutoffs with pseudopotential recommendations. It registers the tool for agents and MCP, invokes it before submission, displays warnings in approval previews, and adds integration and range-validation tests.

Changes

Cutoff validation

Layer / File(s) Summary
Implement cutoff comparison
src/aiida_agents/tools/execution/ranges.py
Adds RangeFinding and check_input_ranges. The tool discovers cutoffs, structures, and pseudopotential families, retrieves recommendations, and reports materially low or high ecutwfc and ecutrho values.
Expose and invoke the tool
src/aiida_agents/tools/execution/__init__.py, src/aiida_agents/tools/__init__.py, src/aiida_agents/agents/execution/__init__.py, src/aiida_agents/mcp/tools/__init__.py, src/aiida_agents/agents/execution/prompt.md, tests/agents/execution/test_integration.py
Exports and registers check_input_ranges as a read-only execution tool. The execution prompt requires cutoff checks before submission and defines result interpretation.
Display warnings and validate coverage
src/aiida_agents/cli/hitl.py, tests/tools/execution/test_ranges.py, pyproject.toml
Approval previews display range findings. Tests cover cutoff comparisons, input layouts, family resolution, missing data, and malformed values. The test dependency and mypy configuration support pseudopotential tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ExecutionAgent
  participant check_input_ranges
  participant PseudopotentialFamily
  participant ApprovalPreview
  ExecutionAgent->>check_input_ranges: validate WorkflowSpec cutoffs
  check_input_ranges->>PseudopotentialFamily: retrieve cutoff recommendations
  PseudopotentialFamily-->>check_input_ranges: return recommendations
  check_input_ranges-->>ExecutionAgent: return RangeFinding results
  ExecutionAgent->>ApprovalPreview: render resolved inputs and warnings
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 70.83% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description directly explains the new cutoff range-checking tool, its integrations, warning behavior, and tests.
Title check ✅ Passed The title clearly and concisely describes the main change: checking cutoffs against pseudopotential recommendations.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/aiida_agents/cli/hitl.py (1)

132-151: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider a direct test asserting the approval preview surfaces a range finding.

_range_warnings and its use in _print_previews (Line 173) are new, user-visible safety behavior for the approval flow. tests/tools/execution/test_ranges.py thoroughly covers check_input_ranges itself, but none of the files in this review exercise _print_previews/_range_warnings directly (e.g., asserting that a submission preview with an out-of-range cutoff prints the ⚠️ line). If such a test exists elsewhere it is out of scope here; otherwise, add one so a future refactor of _submission_args or _print_previews cannot silently drop the warning without a test failing.

Also applies to: 173-174

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/cli/hitl.py` around lines 132 - 151, Add a focused test
covering the approval preview path through _print_previews and _range_warnings:
build a submission with an out-of-range cutoff, invoke the preview, and assert
the output includes the warning line marked with ⚠️. Reuse existing test helpers
and preserve the current check_input_ranges behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiida_agents/tools/execution/ranges.py`:
- Around line 121-143: Update _cutoffs_in to retain every valid cutoff
occurrence rather than discarding later values with the key-in-found guard; use
a collection that preserves each path and numeric value. Adjust
check_input_ranges to iterate through all returned occurrences and evaluate each
independently, while retaining deterministic ordering and existing invalid-value
handling.

---

Nitpick comments:
In `@src/aiida_agents/cli/hitl.py`:
- Around line 132-151: Add a focused test covering the approval preview path
through _print_previews and _range_warnings: build a submission with an
out-of-range cutoff, invoke the preview, and assert the output includes the
warning line marked with ⚠️. Reuse existing test helpers and preserve the
current check_input_ranges behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 36efe03c-37ed-4432-a1b2-55e2e4382713

📥 Commits

Reviewing files that changed from the base of the PR and between ed3c010 and 12e43db.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (10)
  • pyproject.toml
  • src/aiida_agents/agents/execution/__init__.py
  • src/aiida_agents/agents/execution/prompt.md
  • src/aiida_agents/cli/hitl.py
  • src/aiida_agents/mcp/tools/__init__.py
  • src/aiida_agents/tools/__init__.py
  • src/aiida_agents/tools/execution/__init__.py
  • src/aiida_agents/tools/execution/ranges.py
  • tests/agents/execution/test_integration.py
  • tests/tools/execution/test_ranges.py

Comment on lines +121 to +143
def _cutoffs_in(inputs: dict[str, t.Any]) -> dict[str, tuple[str, float]]:
"""Cutoffs found anywhere in the spec, keyed by name.

Returns the first of each: a spec that sets ``ecutwfc`` in two namespaces
is describing two sub-calculations, and reporting the same finding twice
would clutter an approval prompt without adding anything.
"""
found: dict[str, tuple[str, float]] = {}
for params_path, params in _find_leaf(inputs, "parameters"):
if not isinstance(params, dict):
continue
for card in _SYSTEM_CARDS:
section = params.get(card)
if not isinstance(section, dict):
continue
for key in _CUTOFF_KEYS:
if key in found or key not in section:
continue
try:
found[key] = (f"{params_path}.{card}.{key}", float(section[key]))
except (TypeError, ValueError):
logger.debug("non-numeric %s at %s", key, params_path)
return found

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

_cutoffs_in checks only the first occurrence of each cutoff key, silently skipping a legitimately different value in a second namespace.

_cutoffs_in returns a dict keyed by cutoff name and keeps only the first (path, value) per key (Line 137: if key in found or key not in section: continue). The docstring justifies this by assuming a duplicate is "describing two sub-calculations" reporting "the same finding twice." That assumption does not hold in general: workflows such as PwRelaxWorkChain carry independent parameter namespaces (a relax namespace and a separate final-SCF-style namespace) that can legitimately use different cutoffs. If the first namespace's cutoff matches the recommendation but the second namespace's does not, check_input_ranges never reports it, because check_input_ranges (Line 291) iterates sorted(cutoffs.items()), which has already discarded every occurrence but the first.

This defeats the stated purpose of the check: catching "a cutoff of 12 Ry" that "passes schema validation ... and produces a converged, authoritative, wrong number" (Lines 5-7), specifically for the sub-calculation the tool never looks at.

Return every occurrence instead of the first, and check each one.

🐛 Proposed fix to check every cutoff occurrence, not just the first
-def _cutoffs_in(inputs: dict[str, t.Any]) -> dict[str, tuple[str, float]]:
-    """Cutoffs found anywhere in the spec, keyed by name.
-
-    Returns the first of each: a spec that sets ``ecutwfc`` in two namespaces
-    is describing two sub-calculations, and reporting the same finding twice
-    would clutter an approval prompt without adding anything.
-    """
-    found: dict[str, tuple[str, float]] = {}
+def _cutoffs_in(inputs: dict[str, t.Any]) -> list[tuple[str, str, float]]:
+    """Every cutoff found anywhere in the spec, as ``(key, path, value)`` triples.
+
+    All occurrences are returned, not just the first: a workflow such as
+    ``PwRelaxWorkChain`` carries independent namespaces (a relax stage and a
+    final-SCF stage) that can legitimately use different cutoffs, and a bad
+    value in the second one must still be checked.
+    """
+    found: list[tuple[str, str, float]] = []
     for params_path, params in _find_leaf(inputs, "parameters"):
         if not isinstance(params, dict):
             continue
         for card in _SYSTEM_CARDS:
             section = params.get(card)
             if not isinstance(section, dict):
                 continue
             for key in _CUTOFF_KEYS:
-                if key in found or key not in section:
+                if key not in section:
                     continue
                 try:
-                    found[key] = (f"{params_path}.{card}.{key}", float(section[key]))
+                    found.append(
+                        (key, f"{params_path}.{card}.{key}", float(section[key]))
+                    )
                 except (TypeError, ValueError):
                     logger.debug("non-numeric %s at %s", key, params_path)
     return found

And update the caller's loop accordingly:

-    for key, (path, value) in sorted(cutoffs.items()):
+    for key, path, value in sorted(cutoffs):
         expected = recommended.get(key)

Also applies to: 290-322

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida_agents/tools/execution/ranges.py` around lines 121 - 143, Update
_cutoffs_in to retain every valid cutoff occurrence rather than discarding later
values with the key-in-found guard; use a collection that preserves each path
and numeric value. Adjust check_input_ranges to iterate through all returned
occurrences and evaluate each independently, while retaining deterministic
ordering and existing invalid-value handling.

@Jaweria-B
Jaweria-B merged commit ee4ad63 into main Jul 31, 2026
5 checks passed
@Jaweria-B
Jaweria-B deleted the feature/physics-range-validation branch July 31, 2026 10:46
Jaweria-B added a commit that referenced this pull request Jul 31, 2026
The overview's "what is deliberately absent" section said physics-range
validation was deferred and that batch writes had no path. Both shipped --- in
#54 and #57 --- and neither line was updated. This page is the first thing a
reader reaches for, so it was actively telling them two capabilities did not
exist.

Replaced with what is *actually* absent, which in both cases is narrower and
more useful than "not built":

- physics checking exists but does not block, because a low cutoff is
  under-converged for production and reasonable for a smoke test, so refusing
  would be wrong about as often as it was right;
- k-point spacing is unchecked because no installed package publishes a
  recommended one, and inventing a bound is the failure this project is
  built against;
- workflow *construction* is genuinely absent: sequences are composed by
  chaining existing workflows, and nothing emits a new WorkChain class.

The Execution agent's description and the grounding check's both drifted the
same way and are brought up to date with the tools they now hold.
Jaweria-B added a commit that referenced this pull request Jul 31, 2026
* ✨ agents: make the handoff between specialists a typed message

The proposal asked for an agent-to-agent communication protocol. What existed
was a formatted string: the CLI pasted step one's prose answer into step two's
prompt under a heading. It worked, and it lost the thing that mattered most.

A diagnosis says "the failure is in PwCalculation pk 334407". The next step
needs 334407, and with a text handoff the only route to it is a second model
reading that sentence and pulling the number back out. That is a transcription
step between two language models sitting directly in front of a write, and a
resubmission aimed at a mistyped pk is the expensive kind of wrong.

`Handoff` carries two halves: the producing agent's findings verbatim, and the
node references its *tools* returned -- extracted from tool output, not from
the answer, because a pk the agent wrote down may be a typo of one it was given
and a pk taken from the tool return cannot be. The receiving agent is told to
use those identifiers directly rather than re-read them out of the prose.

Three things it deliberately is not.

Not a transport. The protocol is satisfied as a defined message with a
producer, a consumer and typed content, not as a channel between processes.
Both specialists run in one interpreter against one profile, so a transport
would add moving parts for no gain -- and routing a specialist's output
anywhere but the CLI would break the approval guarantee ADR-09 is built around.

Not a summary. The findings pass through unchanged; compressing them would put
a third party in the position of rewriting a conclusion it did not reach.

Not unbounded. References are capped at twenty, because a query matching
hundreds would crowd out the findings they exist to support, and the next step
can always query again.

The REPL extracts from the turn's own messages rather than the accumulated
history: handing a later step pks from an earlier, unrelated question and
calling them "what this step found" would be false.

ADR-09 and the architecture overview are revised to match -- both still
described the handoff as labelled text.

* 📚 architecture: stop claiming two shipped features are absent

The overview's "what is deliberately absent" section said physics-range
validation was deferred and that batch writes had no path. Both shipped --- in
#54 and #57 --- and neither line was updated. This page is the first thing a
reader reaches for, so it was actively telling them two capabilities did not
exist.

Replaced with what is *actually* absent, which in both cases is narrower and
more useful than "not built":

- physics checking exists but does not block, because a low cutoff is
  under-converged for production and reasonable for a smoke test, so refusing
  would be wrong about as often as it was right;
- k-point spacing is unchecked because no installed package publishes a
  recommended one, and inventing a bound is the failure this project is
  built against;
- workflow *construction* is genuinely absent: sequences are composed by
  chaining existing workflows, and nothing emits a new WorkChain class.

The Execution agent's description and the grounding check's both drifted the
same way and are brought up to date with the tools they now hold.

* 📚 rewrite the README for a project that has features

It still said "engineering scaffolding is in place; feature work has not
started", which stopped being true around twenty tools ago. It is the first
thing anyone evaluating this reads, and it was describing an empty repository.

Now: what the thing does, how to install and run it, and what it cannot do.

The capability section is grouped by what a user wants rather than by module,
and each entry says the non-obvious part -- that a diagnosis reports which
remedies the work chain *already tried*, that the approval guarantee lives on
the tool rather than in a prompt, that every reply is checked for quantities no
tool produced. Those are the reasons to use it over a chat window with the docs
pasted in.

The quickstart is verified rather than assumed: every command named exists in
`--help`, and the provider/model example matches `.env.example` instead of
inventing a model string.

A "known limits" section, because the honest ones are short and a reader
finding them out by surprise is worse than a reader who was told. No release.
It composes workflows rather than writing them. Physics checks warn rather than
block. Only Quantum ESPRESSO has been driven against real failures.

The example output is marked as illustrative: its pks, exit code and handler
name are from a real Quantum ESPRESSO diagnosis, but the prose is abridged and
it is not a verbatim transcript.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant