Skip to content

🐛 stop the context tool asserting things it cannot support - #49

Merged
Jaweria-B merged 1 commit into
mainfrom
fix/context-tool-honesty
Jul 29, 2026
Merged

Jaweria-B merged 1 commit into
mainfrom
fix/context-tool-honesty

Conversation

@Jaweria-B

@Jaweria-B Jaweria-B commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #48, which merged before this landed.

An audit for fabricated data found four values query_run_context supplied itself, which a caller could not tell apart from queried ones. The agents' grounding rules forbid exactly this of the model — it was enforced everywhere except in the tool feeding it.

Value What it actually was
structure_type Defaulted to "metallic" and was returned as though asked for. Nothing filters on it — the response even carried a note saying so — and nothing verifies it, so a model could report "for metallic structures, the median ecutwfc is 60" on a qualifier that came from neither the user nor the database. A parameter that does nothing and asserts something is worse than no parameter.
workflow_type Defaulted to aiida.workflows:PwRelaxWorkChain. A caller who omitted it got confident statistics about a workflow they never named — on a profile with 35 entry points, simply a wrong answer delivered well.
recommended_version "The first code found, or any whose label contains pw." That is not a recommendation; the field name asserted an endorsement the tool has no basis for; and it held a label rather than a version. The branch that would have said "Providing schema recommendation" was also unreachable.
recommended_family The one that could have caused real damage. With no families installed it returned the literal string "SSSP/1.3/PBE/efficiency (needs installation)" — a family label for something not in the profile, in a field a model would reasonably pass on as a workflow input.

Changes

All four removed.

workflow_type is now required for past_successful_workflows, raising a ValueError that names the fix — which RetryOnToolError turns into a correction the model can act on, rather than a silent wrong subject.

The pseudos note still gives the aiida-pseudo install sssp … command. That is guidance in prose telling a human what to run, not a field that looks like a usable node reference.

Test plan

  • ruff / ruff format / mypy clean
  • Full suite: 563 passed, 30 skipped
  • Each removed value has a regression test naming what it used to invent
  • The execution prompt no longer documents structure_type, and states that workflow_type is required
  • Two existing tests asserted the presence of structure_type_filter_note — a test pinning a wart. Updated rather than kept.

Why this is a separate PR

#48 was merged while this commit was still local. A merged PR is finished, so this is rebuilt on current main rather than reopened — same content, fresh branch.

An audit for fabricated data found four values query_run_context supplied
itself, which a caller could not tell apart from queried ones. The agents'
grounding rules forbid exactly this of the model; it was enforced everywhere
except in the tool feeding it.

structure_type defaulted to "metallic" and was returned as though asked for.
Nothing filters on it -- the response even carried a note saying so -- and
nothing verifies it, so a model could report "for metallic structures, the
median ecutwfc is 60" on a qualifier that came from neither the user nor the
database. A parameter that does nothing and asserts something is worse than no
parameter; removed.

workflow_type defaulted to "aiida.workflows:PwRelaxWorkChain". A caller who
omitted it got confident statistics about a workflow they never named, which on
a profile with 35 entry points is simply a wrong answer delivered well. Now
required, raising a ValueError that names the fix -- which RetryOnToolError
turns into a correction the model can act on.

recommended_version was "the first code found, or any whose label contains
'pw'". That is not a recommendation, the field name asserted an endorsement the
tool has no basis for, and it held a label rather than a version. Removed; the
note now says plainly that nothing here ranks the codes and points at
list_codes for selection by entry point. The branch that would have said
"Providing schema recommendation" was also unreachable.

recommended_family is the one that could have caused real damage: with no
families installed it returned the literal string
"SSSP/1.3/PBE/efficiency (needs installation)" -- a family label for something
not in the profile, in a field a model would reasonably pass on as a workflow
input. Removed. The note still tells the user the aiida-pseudo command to
install one, which is guidance in prose rather than a usable-looking reference.

Each removal has a regression test naming the value it used to invent.
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Jaweria-B, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 46515c33-cdac-49d9-a618-08b1afb81512

📥 Commits

Reviewing files that changed from the base of the PR and between ac90326 and 40046c5.

📒 Files selected for processing (3)
  • src/aiida_agents/agents/execution/prompt.md
  • src/aiida_agents/tools/run_context.py
  • tests/tools/test_run_context.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jaweria-B
Jaweria-B merged commit 2088933 into main Jul 29, 2026
5 checks passed
@Jaweria-B
Jaweria-B deleted the fix/context-tool-honesty branch July 29, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant