Repository navigation
Testing/aida review testing - #69
Conversation
|
Warning Review limit reached
Next review available in: 41 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (35)
📝 WalkthroughWalkthroughAdded a read-only ChangesDaemon status diagnostics
Query filter validation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant get_daemon_status
participant AiiDA
Agent->>get_daemon_status: request daemon status
get_daemon_status->>AiiDA: inspect daemon and workers
get_daemon_status->>AiiDA: query pending processes
get_daemon_status-->>Agent: return status and diagnosis
Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
src/aiida_agents/tools/daemon.py (1)
38-48: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAvoid materializing every pending process row.
builder.all()loads the complete matching result set only to computelen(rows)and read the first row. Usebuilder.count()for the pending count and a separate query withbuilder.limit(1)orbuilder.first()for the oldest pending process pk.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/tools/daemon.py` around lines 38 - 48, Update _pending_processes to avoid materializing all pending rows: use builder.count() for the total, then perform a separate ordered query limited to one result (via limit(1) or first()) to retrieve the oldest process pk, preserving the existing (count, pk-or-None) return contract.tests/tools/test_daemon.py (1)
121-124: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the
oldest_pending_pkvalue.The non-null assertion does not verify that the tool returns the oldest pending process. A newest pending PK or a constant pending PK would also pass.
Query the expected oldest pending process with the same ordering contract, then assert exact equality with
status["oldest_pending_pk"].🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/tools/test_daemon.py` around lines 121 - 124, Update the test around the pending-status assertions to query the expected oldest pending process using the same ordering contract as the daemon status tool, then assert exact equality with status["oldest_pending_pk"] instead of only checking it is non-null. Preserve the existing pending-count and diagnosis assertions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/gsoc/manual-test-plan.md`:
- Line 151: Add language identifiers to the remaining fenced code blocks in the
manual test plan, specifically the fences near lines 151, 167, 180, and 194. Use
text or the project's standard console-language identifier on each opening fence
so the markdownlint MD040 violations are resolved.
- Around line 119-123: Update the “pending processes” guidance in the manual
test plan to use the locale-consistent spelling “afterward” instead of
“afterwards,” leaving the surrounding instructions unchanged.
- Around line 125-130: Update the daemon restart verification steps around
`verdi daemon start` and the health query so they first require only a running
daemon with the expected worker count, without requiring zero pending processes
immediately. Add a later check for `pending == 0` after previously interrupted
processes have finished.
In `@src/aiida_agents/agents/execution/prompt.md`:
- Around line 222-224: Update the terminated:false guidance to distinguish
daemon health from process state: after get_daemon_status(), treat running with
workers unavailable or zero as unhealthy, quote diagnosis, and recommend verdi
daemon start; otherwise call get_process_status(pk) before describing the
calculation as running. Start the next step only after get_process_status
confirms the required output exists, and preserve the no-resubmission guidance.
In `@src/aiida_agents/tools/daemon.py`:
- Around line 65-77: In the daemon status message logic, add a workers is None
branch before the workers == 0 check. Return a diagnosis that explicitly states
the worker count is unavailable and avoids reporting worker-dependent queue
status as normal; leave the existing zero-worker, pending, and idle branches
unchanged.
In `@tests/tools/analysis/test_query_builder.py`:
- Around line 387-411: Update
test_a_rejected_filter_would_have_changed_the_answer so both count queries are
scoped to query_archive rather than the shared profile: include the fixture’s
group_label alongside the flat entity_type path, retain the existing filter
comparison, and assert each total through query_archive.expect(...) to ensure
only archive records are counted.
---
Nitpick comments:
In `@src/aiida_agents/tools/daemon.py`:
- Around line 38-48: Update _pending_processes to avoid materializing all
pending rows: use builder.count() for the total, then perform a separate ordered
query limited to one result (via limit(1) or first()) to retrieve the oldest
process pk, preserving the existing (count, pk-or-None) return contract.
In `@tests/tools/test_daemon.py`:
- Around line 121-124: Update the test around the pending-status assertions to
query the expected oldest pending process using the same ordering contract as
the daemon status tool, then assert exact equality with
status["oldest_pending_pk"] instead of only checking it is non-null. Preserve
the existing pending-count and diagnosis assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b68ddb36-dcf5-4c55-9f54-4a00ddb82a47
📒 Files selected for processing (16)
README.mddocs/architecture.mddocs/gsoc/manual-test-plan.mdsrc/aiida_agents/agents/analysis/__init__.pysrc/aiida_agents/agents/analysis/prompt.mdsrc/aiida_agents/agents/execution/__init__.pysrc/aiida_agents/agents/execution/prompt.mdsrc/aiida_agents/mcp/tools/__init__.pysrc/aiida_agents/tools/__init__.pysrc/aiida_agents/tools/_types.pysrc/aiida_agents/tools/analysis/query_builder.pysrc/aiida_agents/tools/daemon.pytests/agents/analysis/test_analysis.pytests/agents/execution/test_integration.pytests/tools/analysis/test_query_builder.pytests/tools/test_daemon.py
| - **Good:** it says the daemon is stopped, tells you to run | ||
| `verdi daemon start`, and says the pending processes **resume on their own** | ||
| afterwards. | ||
| - **Flag it if:** it offers to resubmit them (that would run the same work | ||
| twice), or reports "still waiting, check back later" with no explanation. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the locale-consistent spelling afterward.
The LanguageTool check flags afterwards at Line 121. Replace it with afterward.
🧰 Tools
🪛 LanguageTool
[locale-violation] ~121-~121: In American English, ‘afterward’ is the preferred variant. ‘Afterwards’ is more commonly used in British English and other dialects.
Context: ...ing processes resume on their own afterwards. - Flag it if: it offers to resubmi...
(AFTERWARDS_US)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/gsoc/manual-test-plan.md` around lines 119 - 123, Update the “pending
processes” guidance in the manual test plan to use the locale-consistent
spelling “afterward” instead of “afterwards,” leaving the surrounding
instructions unchanged.
Source: Linters/SAST tools
| ```bash | ||
| verdi daemon start | ||
| aiida-agents ask "is the daemon healthy?" | ||
| ``` | ||
|
|
||
| - **Good:** running, worker count, nothing pending. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not require zero pending processes immediately after restart.
The previous verdi daemon stop step leaves non-terminal processes behind. After verdi daemon start, those processes can still be resuming or running. Check that the daemon is running with the expected worker count first. Verify pending == 0 only after the processes finish.
Proposed wording
- **Good:** running, worker count, nothing pending.
+ **Good:** running with the expected worker count. Repeat this check after
+ the pending processes finish, then verify that `pending` reaches zero.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ```bash | |
| verdi daemon start | |
| aiida-agents ask "is the daemon healthy?" | |
| ``` | |
| - **Good:** running, worker count, nothing pending. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/gsoc/manual-test-plan.md` around lines 125 - 130, Update the daemon
restart verification steps around `verdi daemon start` and the health query so
they first require only a running daemon with the expected worker count, without
requiring zero pending processes immediately. Add a later check for `pending ==
0` after previously interrupted processes have finished.
| aiida-agents chat | ||
| ``` | ||
|
|
||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language identifiers to the remaining fenced blocks.
markdownlint-cli2 reports MD040 for the fences at Lines 151, 167, 180, and 194. Add text or the project's standard console-language identifier to each opening fence.
Also applies to: 167-167, 180-180, 194-194
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 151-151: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/gsoc/manual-test-plan.md` at line 151, Add language identifiers to the
remaining fenced code blocks in the manual test plan, specifically the fences
near lines 151, 167, 180, and 194. Use text or the project's standard
console-language identifier on each opening fence so the markdownlint MD040
violations are resolved.
Source: Linters/SAST tools
| if workers == 0: | ||
| return ( | ||
| "The daemon is running but has 0 workers, so no process can be picked " | ||
| "up. Run `verdi daemon incr 1`." | ||
| ) | ||
| if pending: | ||
| return ( | ||
| f"The daemon is running with {workers} worker(s) and {pending} " | ||
| "process(es) are in a non-terminal state. That is the normal picture " | ||
| "for work in flight; a process that stays put across several minutes " | ||
| "is worth looking at with get_process_report()." | ||
| ) | ||
| return f"The daemon is running with {workers} worker(s) and nothing is pending." |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Handle an unavailable worker count explicitly.
When get_numprocesses() fails, workers remains None. The pending branches then produce text such as "with None worker(s)". The diagnosis can also describe the queue as normal while the worker state is unknown. Add a workers is None branch before workers == 0.
Proposed fix
if not running:
...
+ if workers is None:
+ return (
+ f"The daemon is running, but its worker count is unavailable. "
+ f"{pending} process(es) are in a non-terminal state."
+ )
if workers == 0:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if workers == 0: | |
| return ( | |
| "The daemon is running but has 0 workers, so no process can be picked " | |
| "up. Run `verdi daemon incr 1`." | |
| ) | |
| if pending: | |
| return ( | |
| f"The daemon is running with {workers} worker(s) and {pending} " | |
| "process(es) are in a non-terminal state. That is the normal picture " | |
| "for work in flight; a process that stays put across several minutes " | |
| "is worth looking at with get_process_report()." | |
| ) | |
| return f"The daemon is running with {workers} worker(s) and nothing is pending." | |
| if workers is None: | |
| return ( | |
| f"The daemon is running, but its worker count is unavailable. " | |
| f"{pending} process(es) are in a non-terminal state." | |
| ) | |
| if workers == 0: | |
| return ( | |
| "The daemon is running but has 0 workers, so no process can be picked " | |
| "up. Run `verdi daemon incr 1`." | |
| ) | |
| if pending: | |
| return ( | |
| f"The daemon is running with {workers} worker(s) and {pending} " | |
| "process(es) are in a non-terminal state. That is the normal picture " | |
| "for work in flight; a process that stays put across several minutes " | |
| "is worth looking at with get_process_report()." | |
| ) | |
| return f"The daemon is running with {workers} worker(s) and nothing is pending." |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/tools/daemon.py` around lines 65 - 77, In the daemon status
message logic, add a workers is None branch before the workers == 0 check.
Return a diagnosis that explicitly states the worker count is unavailable and
avoids reporting worker-dependent queue status as normal; leave the existing
zero-worker, pending, and idle branches unchanged.
| def test_a_rejected_filter_would_have_changed_the_answer( | ||
| query_archive: QueryArchive, | ||
| ) -> None: | ||
| """The dropped filter was not cosmetic: it selects a strict subset. | ||
|
|
||
| Guards the test above from becoming vacuous -- if the filtered and | ||
| unfiltered totals were equal, rejecting the bad shape would prove nothing. | ||
| """ | ||
| both = {"path": [{"entity_type": CALC, "tag": "calc"}], "count_only": True} | ||
| unfiltered = query_nodes(QuerySpec.model_validate(both))["total"] | ||
| filtered = query_nodes( | ||
| QuerySpec.model_validate( | ||
| both | ||
| | { | ||
| "filters": { | ||
| "calc": { | ||
| "field": "attributes.exit_status", | ||
| "operator": "==", | ||
| "value": 0, | ||
| } | ||
| } | ||
| } | ||
| ) | ||
| )["total"] | ||
| assert 0 < filtered < unfiltered |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Scope both count queries to query_archive.
Line 395 builds a CalcJobNode query without group_label. The query_archive fixture is module-scoped and additive. Its contract requires count assertions to scope queries to the fixture's group, not the shared profile. (raw.githubusercontent.com)
Both totals can include unrelated CalcJobNode rows. Other tests can change the counts, or this regression can pass for records outside the archive. Use the flat entity_type plus group_label form with a bare filter, or add an explicit group join. Assert the totals with query_archive.expect(...).
Proposed scoped test
- both = {"path": [{"entity_type": CALC, "tag": "calc"}], "count_only": True}
+ both = {
+ "entity_type": CALC,
+ "group_label": query_archive.group_label,
+ "count_only": True,
+ }
- "filters": {
- "calc": {
- "field": "attributes.exit_status",
- "operator": "==",
- "value": 0,
- }
- }
+ "filters": {
+ "field": "attributes.exit_status",
+ "operator": "==",
+ "value": 0,
+ }
- assert 0 < filtered < unfiltered
+ assert unfiltered == query_archive.expect()
+ assert filtered == query_archive.expect(lambda row: row.exit_status == 0)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_a_rejected_filter_would_have_changed_the_answer( | |
| query_archive: QueryArchive, | |
| ) -> None: | |
| """The dropped filter was not cosmetic: it selects a strict subset. | |
| Guards the test above from becoming vacuous -- if the filtered and | |
| unfiltered totals were equal, rejecting the bad shape would prove nothing. | |
| """ | |
| both = {"path": [{"entity_type": CALC, "tag": "calc"}], "count_only": True} | |
| unfiltered = query_nodes(QuerySpec.model_validate(both))["total"] | |
| filtered = query_nodes( | |
| QuerySpec.model_validate( | |
| both | |
| | { | |
| "filters": { | |
| "calc": { | |
| "field": "attributes.exit_status", | |
| "operator": "==", | |
| "value": 0, | |
| } | |
| } | |
| } | |
| ) | |
| )["total"] | |
| assert 0 < filtered < unfiltered | |
| def test_a_rejected_filter_would_have_changed_the_answer( | |
| query_archive: QueryArchive, | |
| ) -> None: | |
| """The dropped filter was not cosmetic: it selects a strict subset. | |
| Guards the test above from becoming vacuous -- if the filtered and | |
| unfiltered totals were equal, rejecting the bad shape would prove nothing. | |
| """ | |
| both = { | |
| "entity_type": CALC, | |
| "group_label": query_archive.group_label, | |
| "count_only": True, | |
| } | |
| unfiltered = query_nodes(QuerySpec.model_validate(both))["total"] | |
| filtered = query_nodes( | |
| QuerySpec.model_validate( | |
| both | |
| | { | |
| "filters": { | |
| "field": "attributes.exit_status", | |
| "operator": "==", | |
| "value": 0, | |
| } | |
| } | |
| ) | |
| )["total"] | |
| assert unfiltered == query_archive.expect() | |
| assert filtered == query_archive.expect(lambda row: row.exit_status == 0) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/tools/analysis/test_query_builder.py` around lines 387 - 411, Update
test_a_rejected_filter_would_have_changed_the_answer so both count queries are
scoped to query_archive rather than the shared profile: include the fixture’s
group_label alongside the flat entity_type path, retain the existing filter
comparison, and assert each total through query_archive.expect(...) to ensure
only archive records are counted.
Every field of `FilterGroup` had a default and extra keys were ignored, so
*any* mapping validated as an empty group. A per-tag filter the model got
slightly wrong -- `{"exit_status": 410}` rather than the canonical
`{"field": ..., "operator": ..., "value": ...}` -- therefore lowered to
`{"and": []}`, which constrains nothing, and the query returned the entire
unfiltered table.
The flat shorthand was already safe: `_is_filter_tree` keys off `field`/`logic`,
so an unrecognised mapping there is treated as a `{tag: tree}` mapping and
fails. Only the path form, where the tag is valid and the *value* is the
malformed tree, silently widened.
That total is indistinguishable from a real answer, and the grounding check
cannot catch it -- the number genuinely came from tool output. Measured on a
profile of 8 `ArithmeticAddCalculation`s, 3 of them failed: "how many failed?"
returned 8.
`extra="forbid"` on both filter models turns the unparseable shape into a
validation error, which the agent surfaces as a retry, and an empty group is
rejected outright with a message naming the canonical form. Same reasoning as
`_validate_group_labels`: a filter that did not apply must not be
indistinguishable from one that matched everything.
…is down
A submitted process whose daemon is stopped stays in `created`/`waiting`
indefinitely, and nothing about the process itself looks wrong -- it has not
failed, so there is no exit code to explain and no report line to quote.
Measured against a live profile with the daemon stopped, all four existing
tools describe that state correctly and unhelpfully:
get_process_status -> "waiting"
diagnose_process_failure -> failed: false, every field empty
get_process_report -> the last line it logged before stopping
wait_for_process -> terminated: false, after the full timeout
None of them can say why, so the agent's honest answer is "it is still
running, check back later" -- forever. "My calculations are not running" is
among the most common things an AiiDA user asks, and the system was blind to
it.
`get_daemon_status` supplies the missing fact: whether the daemon runs, how
many workers it has, how many processes are in a non-terminal state, and a
`diagnosis` naming the fix. It is read-only -- it never starts, stops or
resizes the daemon, because a daemon the user did not start is not one an
agent should start for them.
It sits at the shared `tools/` top level rather than under `tools/analysis/`,
per that package's rule: both agents need it. Analysis explains a stuck
process; Execution hits the same state when `wait_for_process` runs out.
The execution prompt previously told the model that a `terminated: false`
wait is "not a failure and not a hang". That is true only while the daemon
runs, so it now branches on `get_daemon_status()` and, for a stopped daemon,
says the process resumes on its own -- resubmitting would run the same work
twice.
The automated suite covers units and wiring; what it cannot cover is whether the assembled system answers a real question correctly against a real profile. This is that pass, ordered the way a user meets the CLI. Each phase records what a good answer looks like and what to flag, because the failure mode that matters here is not a crash -- it is a fluent, well-formatted wrong answer. Hence the emphasis throughout on checking the tool calls behind a reply and reconciling its numbers against `verdi`.
The docs leaned on the em-dash as a general-purpose connector: 218 of them across 19 files, often several to a paragraph. It reads as a tic rather than as punctuation, and it is the main thing making the prose sound machine-written rather than developer-written. Each one is now a colon, comma, full stop or parenthesis, chosen per sentence. No claim, number or recommendation changed; this is punctuation and sentence boundaries only. The agent system prompts are included, and not only for consistency. A model takes its register from its prompt, so the em-dashes in `prompt.md` were turning up in the answers the agents give users. `analysis/prompt.md` also used the ASCII ` -- ` for the same purpose and is normalised with the rest. CLI flags (`--profile`, `--agent`) are untouched, as are ordinary hyphenated compounds like read-only and approval-gated. Two doubled-up headings that the rewrite exposed are also fixed: ADR-05 and ADR-07 had a colon in the title and another introducing the subtitle.
9c48465 to
ce03512
Compare
No description provided.