Repository navigation
✨ cli: add Click subcommands and override flags - #24
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24 +/- ##
=======================================
Coverage ? 91.52%
=======================================
Files ? 39
Lines ? 1651
Branches ? 181
=======================================
Hits ? 1511
Misses ? 114
Partials ? 26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Replace the bare REPL entry point with a Click group so the CLI is scriptable, not just interactive (issue aiidateam#7): `ask` (one-shot), `chat` (the REPL, still the default with no subcommand), `check` (probe and warm the model), `config show`, and `rag build`. Global `--provider` / `--model` / `--profile` override env and `.env` via pydantic-settings' init-kwarg precedence. A missing local Ollama model (chat or embedding) becomes a one-time pull prompt, and `rag build` offers to install the docs toolchain, shows an embed progress bar, and reports failures as clean errors, not tracebacks. The CLI moves into a `cli/` package split by concern (commands, repl, hitl, session, ollama, config, output); help renders via rich-click, `-h` is a help alias tree-wide, and heavy aiida imports stay lazy so `--help` never loads AiiDA. On the RAG side, `index_docs()` gained a progress callback and no longer skips an empty leftover collection as "already built".
744f114 to
7d8ef95
Compare
Follow-ups from the PR aiidateam#24 review, all UX polish: - `config show` gains a Group column, so the two `base_url` fields (provider vs Ollama) stop sharing one ambiguous label; its Env-var column now folds instead of truncating, keeping a long name copy-pasteable on a narrow terminal. - `config show` also runs the unrecognised-settings check, so a typo'd `AIIDA_AGENTS_*` key surfaces there too, matching `check`, `chat`, and `ask`. - `index_docs` returns an `IndexOutcome`, so `rag build` reports "already built", "left unchanged", or "ready" rather than always claiming a fresh build after skipping an up-to-date index. - Fix a stale example in `dev/setup_test_profile.py` that imported the removed `cli.main`; it now uses `aiida-agents --profile chat`.
More follow-ups from the PR aiidateam#24 review, all UX/tidiness: - `config show`'s Value column now folds instead of ellipsizing, so a long URL or path (`http://localhost:11434/v1`, the vector-db path) stays fully readable at 80 cols, matching the Env var column. - `rag build` renders its progress bar transiently, so re-running on an already-built index no longer leaves a stale, never-updated "Cloning docs" frame above the "already built" line. The permanent success lines are console prints above the bar, so they survive. - Move the unrecognised-settings typo check into the `cli` group callback, replacing the three scattered per-command calls. Every subcommand now flags a typo'd `AIIDA_AGENTS_*` key once, including `rag build`, which was previously uncovered. - Note `IndexOutcome` in the `rag` package's Public API docstring.
`warn_on_unrecognized_settings` only inspected keys already inside the `AIIDA_AGENTS_` namespace, so a suffix typo (`AIIDA_AGENTS_PROVDER`) was flagged but a prefix typo (`AIIDA_AGENS_PROVIDER`) fell outside the scan and was dropped silently, leaving the setting at its default. Also scan out-of-namespace keys and, when one closely resembles a real setting name (difflib ratio >= 0.9), flag it with a "did you mean X?" suggestion. The threshold clears genuine near-misses (~0.97, a one- or two-character slip in a 15+ char name) while leaving real neighbours like `OPENAI_API_BASE` (~0.8, next to `OPENAI_API_KEY`) alone. Both the process environment and `.env` are covered.
Rounds out the command surface, separates checking from warming, and makes a mistyped setting fatal. New commands: `--version`; `doctor` (profile / model / RAG / docs toolchain checks); `mcp serve` (the existing MCP server, under one binary); `rag status` (a network-free store snapshot), `rag search` (query the index directly), and `rag clear`; `config init` (scaffold a correct .env, derived from the settings so it can't teach a typo) and `config path`. `check` no longer warms the model: it verifies config, endpoint reachability, and model availability with one cheap listing under a timeout (no generation, so it is fast and can't hang). The generation that pre-loads a local model moves to a new `warm` command. A mistyped key like `AIIDA_AGENS_PROVIDER` was dropped silently, so a command ran with an unintended default. Detection moves to `find_unrecognized_settings`, and a `_needs_recognized_settings` guard exits 2 before a command acts. `--help`, completion, and `config` commands are unaffected, so the problem stays diagnosable. Every settings field gains an attribute docstring, surfaced as its description (`use_attribute_docstrings`), so IDE hover/autocomplete, `config init`, and `config show` document it from one source; the `config init` template now says, per API key, which provider it is for. New helpers (`store.index_status`, `config._env_template`, `session._check_reachable`) and the two previously-uncovered paths (`_dotenv_keys`, `_parse_args`) are tested.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe legacy CLI entrypoint is replaced by a Click-based command package with agent, configuration, diagnostics, MCP, Ollama, RAG, and interactive REPL commands. Settings validation, RAG status reporting, logging, public exports, and associated test coverage are expanded. ChangesCLI and settings
Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Review follow-ups: make the config surface complete and consistent, and stop `doctor` warming the model. `config init` and `config show` each hardcoded their own setting list, so they drifted: `config show` omitted the logging settings, and both hid `AIIDA_AGENTS_PORT` and `AIIDA_AGENTS_TOOL_RETRIES`. Both now use `_SETTINGS_GROUPS` (the same source the typo-detector uses), so a new field surfaces in all three at once; tests pin both to `== _known_env_var_names()`. `doctor`'s model check used the generation probe, so it loaded and warmed the model and was slow. Factored `_probe_reachable` (lists the endpoint's models, no generation); `check` and `doctor` share it, and `_probe_model` is now `warm`-only. Also: `check` treats a model missing from `/models` as fatal only for Ollama (its listing is authoritative), a warning for cloud / openai-compatible (whose listing may be partial); `_run_diagnostics` is typed `ModelSettings` (two `type: ignore` gone); and a merely-absent key now reads "API key not set" rather than "Authentication failed".
Review follow-ups from the end-to-end pass at 32f86ea: two real bugs in the diagnostics/reachability path, plus CLI-output polish. doctor aborted with an uncaught IndexError when a health check raised a message-less exception: the detail column did str(exc).splitlines()[0], and "".splitlines() is [], so [0] threw and killed the whole report (which is meant to isolate each check). Extracted _short_reason(exc) (first non-empty line, capped at 100 chars, "" when empty) and use it for all three checks; pinned by test_short_reason_summarizes_exception. The reachability timeout caught the builtin TimeoutError, but on Python 3.10 (our minimum) asyncio.wait_for raises the distinct asyncio.TimeoutError (they only became aliases in 3.11), so on 3.10 the timeout escaped uncaught, defeating the "never hang" guard and crashing doctor via the bug above. Now catches asyncio.TimeoutError. Polish: every user-facing print() in the REPL and HITL flow is now click.echo (ruff T20 now enforces it; the pre-existing carve-outs stay, staged for a later broad-select pass); the HITL confirm uses click.confirm instead of a hand-rolled input() (EOF-safe); config show renders a None value as "unset" like a secret (was "(unset)"); and the two .env key parsers are unified into _settings._dotenv_keys.
The cli package already splits by concern (repl, hitl, session, ollama,
output); commands.py was the outlier, holding the root group plus the
config/rag/mcp groups and their helpers in one ~590-line file. Split
each group into its own module, leaving commands.py as a thin assembly
root.
- _guards.py (new): the shared fail-fast typo guard
(_needs_recognized_settings, _report_setting_problems), so the group
modules use it without importing commands.py (no cycle).
- config.py: gains the config group (show/path/init), beside the
_config_rows / _env_template data it renders.
- rag.py (new): the rag group (build/status/search/clear) and the
docs-toolchain helpers (_module_missing / _pip_install /
_ensure_docs_toolchain).
- mcp.py (new): the mcp group (serve).
- commands.py: now just the root group, the global overrides, the core
commands (chat/ask/check/warm/doctor), and cli.add_command() for the
three subgroups.
Subgroups switch from @cli.group() to standalone @click.group("name")
registered on the root, so imports flow one way (groups -> _guards,
commands -> groups). No behavior change: same command tree, same -h
alias, 260 passed; the tests' monkeypatch paths follow the moved code.
Ctrl-C / Ctrl-D at the write-approval prompt raised click.Abort, which Click turned into "Aborted!" and tore down the whole REPL session. That is inconsistent with Ctrl-C in the main loop (which just re-prompts) and surprising at a y/N confirm, where the safe reading is "cancel this submission." Catch click.Abort and treat it as a decline, so the submission is cancelled and the chat session continues.
check / warm / doctor built settings with _resolve_model_settings before their try, and config show built them inside _config_rows, so an invalid AIIDA_AGENTS_* value (bad provider, max_tokens=0, max_tokens >= context_length, ...) surfaced as a raw pydantic ValidationError traceback. config show was the worst: the command you'd run to debug config crashed on the bad config. chat/ask were already clean (they go through _build_agent). - _settings._format_validation_error: renders a ValidationError as concise "field: reason" lines, dropping pydantic's docs URL and the "Value error," prefix. - session._resolve_settings_or_fail: converts a ValidationError to a ClickException; check/warm/doctor and _build_agent use it. - config._config_rows is now resilient per group: a bad group renders its fields as "(invalid: <reason>)" and the healthy groups still show, so config show never crashes. - config init catches OSError (e.g. --path into a missing directory) and raises a clean "Could not write <path>" instead of a FileNotFoundError traceback. Regression tests: clean error from check on a bad value, config show survives a bad group, config init on a missing dir, and the formatter's concise output.
Testing-infra pass over tests/test_cli.py: less duplication, stronger assertions, fewer mocks. No production code changed. - Parametrize the check/warm probe split (two tests into one that pins check never calling the generation probe) and _ollama_model_names (two into one). Parametrize the -h alias across nesting levels (was four asserts in one body). - Strengthen assertions: config show now checks the table rendered real values (was exit-0 only); -h asserts help actually rendered; the help-bypasses-guard test asserts the guard never fired (no "did you mean"), not just exit 0. - Drop a mock: the missing-API-key test stubbed aiida.load_profile; the autouse test profile makes that real, so it mocks only the external step (get_agent, raising as pydantic-ai would for a missing key). Left tests/test_settings.py and tests/rag/test_indexing.py as-is: the former is already parametrized by behavior across the settings groups, the latter tests distinct scenarios against a real ChromaDB client (mocking only the slow network-bound clone).
The provider list lived in three places that could drift: the _Provider Literal (the real source), the --provider help text, and the field docstring. Make the Literal the single source. - _settings._choices() extracts a Literal alias's values; _PROVIDER_CHOICES = _choices(_Provider). - The --provider flag is now click.Choice(_PROVIDER_CHOICES, case_sensitive=False): --help renders the choices, a bad value is a clean "invalid choice" usage error, and the help string no longer restates the list. (Env / .env values are still validated by pydantic.) - get_model's fallback is assert_never(cfg.provider): a new provider without a branch fails mypy, not at runtime. - The choice-field docstrings (provider/embed_backend/log_level) still enumerate their values for config init / IDE hover, but a new drift test asserts each lists every Literal value, so they can't go stale.
The bodies of the ``rag status`` and ``rag clear`` commands were only ever reached by the fail-fast typo guard and ``-h``, never exercised, so their rendering and confirm branches were untested (``rag.py`` at 38%). Add six CLI-level tests against the real command paths: - ``rag status`` on an unbuilt store (reports not-built, points to ``rag build``) and on a populated one (renders the collection row and its build metadata from a real chroma collection). - ``rag clear`` for the no-store no-op, the ``--yes`` bypass, and the interactive ``y`` (deletes) / ``n`` (keeps) confirm branches. Each test drives ``RagSettings``' own persistence path via the env var instead of mocking ``index_status``, so the store is real on disk.
A polishing pass, batched into one commit:
- logging: the console handler no longer special-cases INFO (which hid
INFO despite the field documenting it as the console level). It now
honours log_level directly, and the default is WARNING so a session
stays uncluttered; the log file still captures the full tool-call /
reply traces regardless, since those log at DEBUG.
- settings: name the env namespace and .env filename once (_ENV_PREFIX /
_ENV_FILE) and reuse them in the typo helpers, dropping the
.get("env_prefix", "") whose "" fallback could misclassify every key.
Replace the opaque (str, str | None) tuple with a SettingProblem
NamedTuple, and split the classifier into a pure
_classify_present_keys(present_keys) core behind the
find_unrecognized_settings I/O boundary. _guards now documents the
surface-agnostic-detection vs CLI-adapter layering.
- output: check colors its status marks (green check / red cross /
yellow bang) to match doctor, and every ellipsis is spelled as three
ASCII dots.
- deps: leave click unbounded (>=8, reached only through rich_click)
rather than a speculative cap; the lockfile pins the exact version.
Batched structure/typing polish of the CLI layer: - cli/session.py -> cli/agent.py. The module builds, runs, and probes the agent; it holds no session state (the interactive session is repl.py), so "agent" describes it and "session" misled. The `ask` coroutine and the `ask_cmd` command (exposed as `ask`) are unchanged. - name the opaque positional-tuple returns as NamedTuples, matching the codebase's existing records (SettingProblem, CollectionInfo): ConfigRow (config show), DiagnosticRow (doctor), Reachability (the reachability probe), and hitl._Preview (was a bare type alias). Consumers unpack unchanged; the config-row tests now use named access. - add a "Public API" docstring section to the package-root and immediate subpackage __init__ files; drop the stray one from agents/analysis (level 3, internal). - config show folds the Setting column so a long field name (e.g. openrouter_api_key) is no longer rich-truncated at 80 cols. - pin the console-honours-log_level policy with a parametrized test.
tests/test_cli.py was a flat 646-line module covering the whole cli/ package. Split it into tests/cli/test_<module>.py mirroring the src layout (test_agent, test_commands, test_config, test_guards, test_ollama, test_output, test_rag, test_repl), each with its own imports and helpers. Also: - move tests/validator/test_hitl.py to tests/cli/test_hitl.py (hitl is a cli module) and fold in the _parse_args test; drop the validator/ dir. - move the two _dotenv_keys tests to tests/test_settings.py (they test _settings, not the cli). Pure move: no test added or removed (280 passed / 5 skipped, unchanged).
Review-driven polish of the cli layer, all behaviour-preserving: - underscore the module-private record tuples (_ConfigRow / _DiagnosticRow / _Reachability), matching hitl._Preview: a record's visibility now follows its producing function's (public SettingProblem / IndexStatus stay un-prefixed). - type config show's Source column as a Literal (_ValueSource = "flag"/"env"/".env"/"default") instead of bare str. - single-source the openai-compatible api_key sentinel as _API_KEY_UNSET, shared by the field default and config show's set/unset masking (was the bare string "api-key-not-set" duplicated across the two modules). - _build_agent takes a resolved ModelSettings and returns just the Agent, instead of taking loose provider/model strings and returning an (agent, settings) tuple whose second half `ask` discarded. Each command resolves settings once (the str->settings boundary) and threads them to both _build_agent and the REPL. - give `doctor` its own self-contained module: move the command and its checks (_run_diagnostics, _short_reason, _DiagnosticRow) out of commands.py into cli/doctor.py, registered like the config/rag/mcp groups. doctor was the one command with real logic (cross-cutting profile / model / RAG / docs checks) rather than thin wiring; commands.py keeps the root group and the thin chat/ask/check/warm bodies.
`check` (_check_reachable) and `doctor` (_run_diagnostics) each hand-rolled the same three-way decision: model advertised -> available; absent on Ollama -> not pulled (fatal); absent elsewhere -> unlisted, may still work. The rendering differs (echo + exit vs a table row) but the decision was one policy duplicated in two places, so a change had to move in lockstep. Extract `_model_availability(reach, provider) -> _ModelAvailability` (a Literal TypeAlias) beside `_probe_reachable`; both callers now dispatch on it and keep their own rendering, and each closes with `assert_never` so a newly added provider variant is a compile-time error in both (matching the provider dispatch in agents/_models.py). The "Ollama's /models listing is authoritative, a cloud endpoint's may be partial" rule now lives in exactly one place.
|
@coderabbitai help |
ChatThere are 3 ways to chat with CodeRabbit:
CodeRabbit commands
Other keywords and placeholders
CodeRabbit configuration file (
|
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (2)
src/aiida_agents/cli/repl.py (1)
96-170: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider logging swallowed REPL exceptions.
The blind
except Exceptionat Line 154 (flagged by Ruff BLE001) is reasonable for keeping the REPL alive, but the exception is only echoed to the console, never recorded to the trace log configured in_configure_logging. A transient bug surfaced this way leaves no trace for later debugging.♻️ Optional: log before echoing
+import logging ... except Exception as exc: + logging.getLogger(__name__).exception("REPL turn failed") click.echo(f"❌ Error: {exc}") continue🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/cli/repl.py` around lines 96 - 170, Update the broad exception handler in _run_repl to record the caught exception through the trace logger configured by _configure_logging before echoing the error and continuing the REPL. Preserve the existing keep-alive behavior and user-facing error output.src/aiida_agents/cli/commands.py (1)
108-143: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate probe/diagnose/exit block in
checkandwarm.Both commands repeat the identical
try: <probe>(settings) / except Exception as exc: _diagnose_probe_failure(...); raise SystemExit(1) from excblock. Worth extracting to avoid the two copies drifting apart later.♻️ Suggested consolidation
+def _run_probe_or_fail(probe, settings) -> None: + try: + probe(settings) + except Exception as exc: + _diagnose_probe_failure(settings, exc) + raise SystemExit(1) from exc + + `@cli.command`() `@click.pass_context` `@_needs_recognized_settings` def check(ctx: click.Context) -> None: ... click.echo(f"Checking {settings.provider}:{settings.model} ...") - try: - _check_reachable(settings) - except Exception as exc: - _diagnose_probe_failure(settings, exc) - raise SystemExit(1) from exc + _run_probe_or_fail(_check_reachable, settings) ... `@cli.command`() `@click.pass_context` `@_needs_recognized_settings` def warm(ctx: click.Context) -> None: ... start = time.monotonic() - try: - _probe_model(settings) - except Exception as exc: - _diagnose_probe_failure(settings, exc) - raise SystemExit(1) from exc + _run_probe_or_fail(_probe_model, settings) click.echo(f"✓ warmed in {_format_duration(time.monotonic() - start)}")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiida_agents/cli/commands.py` around lines 108 - 143, Extract the shared probe failure-handling flow from check and warm into a helper that accepts the settings and probe callable, invokes it, diagnoses exceptions via _diagnose_probe_failure, and exits with status 1 while preserving exception chaining. Replace both inline try/except blocks with calls to this helper, passing _check_reachable for check and _probe_model for warm.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/cli/agent.py`:
- Around line 36-47: Update the ask function’s query logging so the raw question
is not written at INFO; move the full prompt logging to DEBUG or log only a safe
truncated/redacted representation while preserving the agent.run and tool-trace
behavior.
In `@src/aiida_agents/cli/hitl.py`:
- Around line 189-250: Add tests covering the approval retry flow through
_handle_deferred and _rerun_with_denials, including re-triaging a corrected
DeferredToolRequests result and stopping after _MAX_APPROVAL_ROUNDS without
submitting. Assert the returned history and submission behavior for both paths,
while preserving the existing cancellation and terminal-result behavior.
In `@src/aiida_agents/cli/mcp.py`:
- Around line 22-35: The mcp_serve command must honor the global profile
override. Accept or obtain the Click context, read the selected profile from
ctx.obj["profile"], and load that profile before invoking server.mcp.run();
preserve the existing port resolution and startup behavior.
In `@src/aiida_agents/cli/ollama.py`:
- Around line 55-58: Update _ollama_pull to make subprocess.run enforce a
successful “ollama pull” exit status instead of ignoring failures, so command
errors propagate immediately while preserving the existing command and output
behavior.
In `@src/aiida_agents/cli/output.py`:
- Around line 44-52: Update _render_part so part.args is rendered as literal
text rather than interpolated into a Rich markup string, matching the escaping
approach used for tool-return content below. Preserve the existing Args label
and formatting while preventing bracketed or closing-tag-like argument values
from being parsed as markup.
In `@tests/cli/test_config.py`:
- Around line 111-124: Update
test_config_init_writes_template_and_refuses_overwrite to accept the existing
monkeypatch fixture and change the process working directory to tmp_path before
invoking config init. Keep the existing assertions and invocation sequence
unchanged so all filesystem operations target the isolated temporary directory.
- Around line 126-131: Update test_config_path_reports_the_env_file to execute
the CLI from tmp_path by changing into that directory before invoking config
path, ensuring it reads an isolated environment file. Keep the existing
assertions, including “not present,” once the test’s working directory is
isolated.
---
Nitpick comments:
In `@src/aiida_agents/cli/commands.py`:
- Around line 108-143: Extract the shared probe failure-handling flow from check
and warm into a helper that accepts the settings and probe callable, invokes it,
diagnoses exceptions via _diagnose_probe_failure, and exits with status 1 while
preserving exception chaining. Replace both inline try/except blocks with calls
to this helper, passing _check_reachable for check and _probe_model for warm.
In `@src/aiida_agents/cli/repl.py`:
- Around line 96-170: Update the broad exception handler in _run_repl to record
the caught exception through the trace logger configured by _configure_logging
before echoing the error and continuing the REPL. Preserve the existing
keep-alive behavior and user-facing error output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b6bdc934-e122-4ba4-b321-67ea92ed3c30
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (43)
dev/setup_test_profile.pypyproject.tomlsrc/aiida_agents/__init__.pysrc/aiida_agents/_logging.pysrc/aiida_agents/_settings.pysrc/aiida_agents/agents/_models.pysrc/aiida_agents/agents/analysis/__init__.pysrc/aiida_agents/cli.pysrc/aiida_agents/cli/__init__.pysrc/aiida_agents/cli/_guards.pysrc/aiida_agents/cli/agent.pysrc/aiida_agents/cli/commands.pysrc/aiida_agents/cli/config.pysrc/aiida_agents/cli/doctor.pysrc/aiida_agents/cli/hitl.pysrc/aiida_agents/cli/mcp.pysrc/aiida_agents/cli/ollama.pysrc/aiida_agents/cli/output.pysrc/aiida_agents/cli/rag.pysrc/aiida_agents/cli/repl.pysrc/aiida_agents/mcp/__init__.pysrc/aiida_agents/rag/__init__.pysrc/aiida_agents/rag/embeddings.pysrc/aiida_agents/rag/indexing.pysrc/aiida_agents/rag/retriever.pysrc/aiida_agents/rag/store.pysrc/aiida_agents/tools/__init__.pysrc/aiida_agents/tools/_types.pytests/cli/__init__.pytests/cli/test_agent.pytests/cli/test_commands.pytests/cli/test_config.pytests/cli/test_doctor.pytests/cli/test_guards.pytests/cli/test_hitl.pytests/cli/test_ollama.pytests/cli/test_output.pytests/cli/test_rag.pytests/cli/test_repl.pytests/rag/test_indexing.pytests/test_logging.pytests/test_settings.pytests/validator/test_hitl.py
💤 Files with no reviewable changes (2)
- tests/validator/test_hitl.py
- src/aiida_agents/cli.py
| def _rerun_with_denials( | ||
| agent: Agent, | ||
| result: Any, | ||
| pending: DeferredToolRequests, | ||
| auto: dict[str, Any], | ||
| ) -> Any: # pragma: no cover | ||
| """Deny the invalid submissions back to the model and re-run so it corrects | ||
| its own inputs. | ||
|
|
||
| No DB write happens on the worker thread here (denied calls are never | ||
| executed), so this is thread-safe. Returns the new agent result, which may | ||
| itself be a ``DeferredToolRequests`` if the correction proposed fresh | ||
| submissions. | ||
| """ | ||
| click.echo("\n⚠️ Inputs were invalid; asking the agent to correct them.") | ||
| rerun = asyncio.run( | ||
| agent.run( | ||
| None, | ||
| message_history=result.all_messages(), | ||
| deferred_tool_results=pending.build_results(approvals=auto), | ||
| ) | ||
| ) | ||
| _log_tool_calls_debug(rerun.new_messages(), console) | ||
| return rerun | ||
|
|
||
|
|
||
| def _handle_deferred( | ||
| agent: Agent, | ||
| result: Any, | ||
| history: list[ModelMessage], | ||
| ) -> list[ModelMessage]: # pragma: no cover | ||
| """Confirm and run pending submissions, denying invalid ones to the model. | ||
|
|
||
| Each round triages the pending approvals (:func:`_triage_submissions`): if any | ||
| reach the user they are confirmed and submitted (terminal); otherwise the | ||
| invalid ones are denied back to the model to correct and the agent re-runs, | ||
| looping until it stops proposing submissions or the retry budget is spent. | ||
| Cancelling or exhausting the budget returns the pre-turn ``history`` unchanged. | ||
| """ | ||
| for _ in range(_MAX_APPROVAL_ROUNDS): | ||
| pending = result.output | ||
| auto, previews = _triage_submissions(pending) | ||
|
|
||
| if previews: | ||
| return _confirm_and_submit(result, pending, auto, previews, history) | ||
|
|
||
| if not auto: | ||
| return history | ||
|
|
||
| try: | ||
| result = _rerun_with_denials(agent, result, pending, auto) | ||
| except Exception as exc: | ||
| click.echo(f"\n❌ Error: {exc}") | ||
| return history | ||
|
|
||
| if not isinstance(result.output, DeferredToolRequests): | ||
| _print_agent(result.output) | ||
| messages: list[ModelMessage] = result.all_messages() | ||
| return messages | ||
|
|
||
| click.echo("\n⚠️ Too many correction rounds; stopping without submitting.") | ||
| return history |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check whether the pragma:no-cover functions are still exercised by tests.
rg -n '_handle_deferred|_rerun_with_denials|_MAX_APPROVAL_ROUNDS' tests/cli/test_hitl.pyRepository: aiidateam/aiida-agents
Length of output: 160
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== source outline =='
ast-grep outline src/aiida_agents/cli/hitl.py --view expanded || true
echo
echo '== source lines around the retry loop =='
sed -n '150,280p' src/aiida_agents/cli/hitl.py | nl -ba
echo
echo '== all hitl-related tests =='
rg -n "handle_deferred|rerun_with_denials|triage_submissions|_MAX_APPROVAL_ROUNDS|DeferredToolRequests|ToolDenied|approval" tests -S
echo
echo '== test file preview =='
if [ -f tests/cli/test_hitl.py ]; then
sed -n '1,260p' tests/cli/test_hitl.py | nl -ba
fiRepository: aiidateam/aiida-agents
Length of output: 669
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== src/aiida_agents/cli/hitl.py excerpt =='
sed -n '170,260p' src/aiida_agents/cli/hitl.py | awk '{printf "%4d %s\n", NR+169, $0}'
echo
echo '== tests mentioning hitl or approval flow =='
rg -n "hitl|approval|DeferredToolRequests|ToolDenied|_triage_submissions|_confirm_and_submit|_rerun_with_denials|_handle_deferred" tests -S
echo
echo '== test file list for cli =='
fd -e py tests/cli || trueRepository: aiidateam/aiida-agents
Length of output: 8127
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== references to _handle_deferred in source =='
rg -n "_handle_deferred" src tests -S
echo
echo '== tests/cli/test_commands.py excerpt =='
sed -n '130,220p' tests/cli/test_commands.py | nl -ba
echo
echo '== tests/cli/test_hitl.py excerpt around the retry-related tests =='
sed -n '200,320p' tests/cli/test_hitl.py | nl -ba
echo
echo '== source callers around the interactive flow =='
rg -n "confirm_and_submit|handle_deferred|rerun_with_denials|triage_submissions|DeferredToolRequests" src/aiida_agents/cli -SRepository: aiidateam/aiida-agents
Length of output: 523
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== repl caller and interactive flow =='
sed -n '130,190p' src/aiida_agents/cli/repl.py | cat -n
echo
echo '== tests mentioning repl or interactive approval =='
rg -n "repl|interactive approval|_handle_deferred|DeferredToolRequests|click.confirm|Abort" tests -S
echo
echo '== tests/cli/test_commands.py relevant excerpt =='
sed -n '150,210p' tests/cli/test_commands.py | cat -n
echo
echo '== tests/cli/test_hitl.py retry-related excerpt =='
sed -n '200,320p' tests/cli/test_hitl.py | cat -nRepository: aiidateam/aiida-agents
Length of output: 8973
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== tests/cli/test_repl.py =='
sed -n '1,260p' tests/cli/test_repl.py | cat -n
echo
echo '== any DeferredToolRequests / approval flow in repl tests =='
rg -n "DeferredToolRequests|_handle_deferred|click.confirm|Abort|approval|interactive" tests/cli/test_repl.py tests/cli/test_hitl.py tests/cli/test_commands.py -S
echo
echo '== callers of _handle_deferred in tests =='
rg -n "_handle_deferred" tests -SRepository: aiidateam/aiida-agents
Length of output: 7808
Add coverage for the approval retry loop
_rerun_with_denials and _handle_deferred are excluded from coverage, and there’s no test that drives the retry path itself. Add a REPL-level or direct unit test for the loop, especially the max-round exhaustion case and a re-triage after a corrected submission.
🧰 Tools
🪛 Ruff (0.15.20)
[warning] 240-240: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/cli/hitl.py` around lines 189 - 250, Add tests covering the
approval retry flow through _handle_deferred and _rerun_with_denials, including
re-triaging a corrected DeferredToolRequests result and stopping after
_MAX_APPROVAL_ROUNDS without submitting. Assert the returned history and
submission behavior for both paths, while preserving the existing cancellation
and terminal-result behavior.
| def _ollama_pull(model: str) -> None: # pragma: no cover | ||
| """Download a model into the local Ollama server.""" | ||
| click.echo(f"Running: ollama pull {model}") | ||
| subprocess.run(["ollama", "pull", model], check=False) # noqa: S607 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file and surrounding context
git ls-files src/aiida_agents/cli/ollama.py
wc -l src/aiida_agents/cli/ollama.py
cat -n src/aiida_agents/cli/ollama.py | sed -n '1,140p'
# Look for Ruff/flake8 configuration that might affect S603/S607 handling
rg -n --hidden --glob '!*.pyc' --glob '!.git' 'S603|S607|ruff|bandit|per-file-ignores|ignore' pyproject.toml setup.cfg tox.ini .ruff.toml .pre-commit-config.yaml .Repository: aiidateam/aiida-agents
Length of output: 18887
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show Ruff configuration around selection/ignores and per-file ignores
sed -n '121,170p' pyproject.tomlRepository: aiidateam/aiida-agents
Length of output: 2309
Surface ollama pull failures. check=False drops the exit status, so a failed pull still appears successful and the model only fails later on first use.
🧰 Tools
🪛 ast-grep (0.44.1)
[error] 57-57: Command coming from incoming request
Context: subprocess.run(["ollama", "pull", model], check=False)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🪛 Ruff (0.15.20)
[error] 58-58: subprocess call: check for execution of untrusted input
(S603)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/aiida_agents/cli/ollama.py` around lines 55 - 58, Update _ollama_pull to
make subprocess.run enforce a successful “ollama pull” exit status instead of
ignoring failures, so command errors propagate immediately while preserving the
existing command and output behavior.
Source: Linters/SAST tools
| def test_config_init_writes_template_and_refuses_overwrite(tmp_path: Path) -> None: | ||
| """`config init` writes a template, then declines to clobber it without --force.""" | ||
| first = CliRunner().invoke(cli, ["config", "init"]) | ||
| assert first.exit_code == 0 | ||
| written = (tmp_path / ".env").read_text(encoding="utf-8") | ||
| assert "AIIDA_AGENTS_PROVIDER=ollama" in written | ||
|
|
||
| second = CliRunner().invoke(cli, ["config", "init"]) | ||
| assert second.exit_code == 1 | ||
| assert "already exists" in second.output | ||
|
|
||
| forced = CliRunner().invoke(cli, ["config", "init", "--force"]) | ||
| assert forced.exit_code == 0 | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Test writes a real .env outside tmp_path; missing chdir isolation.
CliRunner().invoke() doesn't sandbox the filesystem by itself. Without monkeypatch.chdir(tmp_path), config init writes .env into the actual process cwd, not tmp_path, so (tmp_path / ".env").read_text(...) at Line 115 will raise FileNotFoundError instead of exercising the intended assertion — and it leaves a real, uncleaned .env file in the working tree that can leak into other tests/sessions.
🛠️ Proposed fix
-def test_config_init_writes_template_and_refuses_overwrite(tmp_path: Path) -> None:
+def test_config_init_writes_template_and_refuses_overwrite(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
"""`config init` writes a template, then declines to clobber it without --force."""
+ monkeypatch.chdir(tmp_path)
first = CliRunner().invoke(cli, ["config", "init"])📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_config_init_writes_template_and_refuses_overwrite(tmp_path: Path) -> None: | |
| """`config init` writes a template, then declines to clobber it without --force.""" | |
| first = CliRunner().invoke(cli, ["config", "init"]) | |
| assert first.exit_code == 0 | |
| written = (tmp_path / ".env").read_text(encoding="utf-8") | |
| assert "AIIDA_AGENTS_PROVIDER=ollama" in written | |
| second = CliRunner().invoke(cli, ["config", "init"]) | |
| assert second.exit_code == 1 | |
| assert "already exists" in second.output | |
| forced = CliRunner().invoke(cli, ["config", "init", "--force"]) | |
| assert forced.exit_code == 0 | |
| def test_config_init_writes_template_and_refuses_overwrite( | |
| tmp_path: Path, monkeypatch: pytest.MonkeyPatch | |
| ) -> None: | |
| """`config init` writes a template, then declines to clobber it without --force.""" | |
| monkeypatch.chdir(tmp_path) | |
| first = CliRunner().invoke(cli, ["config", "init"]) | |
| assert first.exit_code == 0 | |
| written = (tmp_path / ".env").read_text(encoding="utf-8") | |
| assert "AIIDA_AGENTS_PROVIDER=ollama" in written | |
| second = CliRunner().invoke(cli, ["config", "init"]) | |
| assert second.exit_code == 1 | |
| assert "already exists" in second.output | |
| forced = CliRunner().invoke(cli, ["config", "init", "--force"]) | |
| assert forced.exit_code == 0 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/cli/test_config.py` around lines 111 - 124, Update
test_config_init_writes_template_and_refuses_overwrite to accept the existing
monkeypatch fixture and change the process working directory to tmp_path before
invoking config init. Keep the existing assertions and invocation sequence
unchanged so all filesystem operations target the isolated temporary directory.
| def test_config_path_reports_the_env_file(tmp_path: Path) -> None: | ||
| """`config path` names the .env it reads and whether it is present.""" | ||
| result = CliRunner().invoke(cli, ["config", "path"]) | ||
| assert result.exit_code == 0 | ||
| assert ".env" in result.output | ||
| assert "not present" in result.output |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Same missing chdir(tmp_path) isolation; "not present" assertion is cwd-dependent.
tmp_path is accepted but unused, and config path reads .env relative to the real cwd. The "not present" assertion only holds by accident of the ambient working directory (and breaks if the sibling config init test above leaks a real .env into that same directory).
🛠️ Proposed fix
-def test_config_path_reports_the_env_file(tmp_path: Path) -> None:
+def test_config_path_reports_the_env_file(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
"""`config path` names the .env it reads and whether it is present."""
+ monkeypatch.chdir(tmp_path)
result = CliRunner().invoke(cli, ["config", "path"])📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_config_path_reports_the_env_file(tmp_path: Path) -> None: | |
| """`config path` names the .env it reads and whether it is present.""" | |
| result = CliRunner().invoke(cli, ["config", "path"]) | |
| assert result.exit_code == 0 | |
| assert ".env" in result.output | |
| assert "not present" in result.output | |
| def test_config_path_reports_the_env_file( | |
| tmp_path: Path, monkeypatch: pytest.MonkeyPatch | |
| ) -> None: | |
| """`config path` names the .env it reads and whether it is present.""" | |
| monkeypatch.chdir(tmp_path) | |
| result = CliRunner().invoke(cli, ["config", "path"]) | |
| assert result.exit_code == 0 | |
| assert ".env" in result.output | |
| assert "not present" in result.output |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/cli/test_config.py` around lines 126 - 131, Update
test_config_path_reports_the_env_file to execute the CLI from tmp_path by
changing into that directory before invoking config path, ensuring it reads an
isolated environment file. Keep the existing assertions, including “not
present,” once the test’s working directory is isolated.
The RAG corpus is built from `sphinx-build -b text`, whose output files
carry a `.txt` extension. That extension leaked into the stored `source`
metadata (`howto/query.txt`), so the agent's citations and `rag search`
referenced files users never see: they read the docs as HTML in the
browser or `.rst` in source, never `.txt`.
Strip the extension once at the ingestion boundary (`_load_docs`), so
the stored doc id is `howto/query` and every consumer (agent citations,
`rag search`) renders it clean without per-site changes. The exclusion
set is made extensionless to match, and `_chunk_text` drops a redundant
`with_suffix("")` that would otherwise clip a dotted path component now
that the source arrives extensionless.
Cosmetic metadata only, not a corpus-shape change, so `_CORPUS_FORMAT`
is untouched; refresh an existing index with `rag build --force`.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pyproject.toml (1)
190-209: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winScope the unused-function suppression to the CLI helpers.
reportUnusedFunction = "none"disables diagnostics for every private function and method in the project, not only the cross-module CLI helpers. Pyright and basedpyright define this setting as the unused-private-function diagnostic. (github.com)Prefer targeted suppressions or a CLI-specific configuration so unused private functions elsewhere in
agents,rag, and other packages remain visible.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pyproject.toml` around lines 190 - 209, Scope the reportUnusedFunction suppression to the CLI helpers instead of disabling unused-private-function diagnostics project-wide. Update the [tool.pyright] configuration to use a CLI-specific exclusion or targeted suppressions for the cross-module helpers in cli/, while preserving diagnostics for private functions and methods in agents, rag, and other packages.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/cli/test_output.py`:
- Around line 112-122: Update test_print_reply_raw_when_piped to patch
aiida_agents.cli.output.console with a Console configured as
force_terminal=False before invoking _print_reply. Use the patched output module
state so the raw Markdown assertions are deterministic and independent of the
capture environment.
---
Nitpick comments:
In `@pyproject.toml`:
- Around line 190-209: Scope the reportUnusedFunction suppression to the CLI
helpers instead of disabling unused-private-function diagnostics project-wide.
Update the [tool.pyright] configuration to use a CLI-specific exclusion or
targeted suppressions for the cross-module helpers in cli/, while preserving
diagnostics for private functions and methods in agents, rag, and other
packages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9738a53a-9130-4b1f-91cb-e1a217de0e1d
📒 Files selected for processing (11)
pyproject.tomlsrc/aiida_agents/cli/__init__.pysrc/aiida_agents/cli/commands.pysrc/aiida_agents/cli/config.pysrc/aiida_agents/cli/output.pysrc/aiida_agents/rag/chunking.pytests/cli/test_output.pytests/cli/test_rag.pytests/rag/test_chunking.pytests/rag/test_retriever.pytests/rag/test_search_tool.py
🚧 Files skipped from review as they are similar to previous changes (5)
- src/aiida_agents/cli/init.py
- src/aiida_agents/cli/commands.py
- src/aiida_agents/cli/config.py
- src/aiida_agents/cli/output.py
- tests/cli/test_rag.py
Two review passes (CodeRabbit + a self-review) plus a couple of cognitive-complexity refactors on the CLI layer. Review fixes: - `mcp serve` now honours the global `--profile`: it loads the selected profile before starting the server, whose lifespan otherwise only loads the default when none is active. - `_ollama_pull` warns on a non-zero exit instead of swallowing it, so a failed pull surfaces now rather than 404-ing on first use. - tool-call args are `escape()`d before rich renders them; a dict repr with bracketed values could otherwise be eaten as markup or raise MarkupError (the tool-return Panel already uses Text() for this). - the raw user query logs at DEBUG, not INFO: it can carry secrets or private project detail, so it only reaches a log sink on opt-in. - drop the stale `.txt` from the analysis prompt's citation example, so it matches the extensionless doc-source id the RAG layer now stores. Simplification: - split `doctor._run_diagnostics` into one `_check_*` helper per subsystem (profile / model / RAG / docs); the command iterates rows by named field (`row.ok`) instead of unpacking into loose locals. - lift `config._env_template`'s per-field logic into `_env_field_lines`, flattening the nested loop. Behaviour-preserving apart from the intended fixes; all dogfooded, and the raw-reply test now forces non-terminal mode so it stays deterministic under `pytest -s`.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/aiida_agents/cli/doctor.py`:
- Line 137: Update the doctor output formatting around the suffix construction
to escape both dynamic row.label and row.detail values before inserting them
into markup-enabled Rich strings, while preserving the static style tags as
markup. Ensure bracketed user-provided text displays literally without being
interpreted as Rich tags.
In `@src/aiida_agents/cli/ollama.py`:
- Line 63: Add the Ruff suppression for S603 to the subprocess.run call that
pulls the Ollama model, while preserving the existing S607 suppression. Update
the inline noqa directive on the result assignment so both reported rules are
suppressed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 00174fbe-098b-40ad-aa16-bb315b19d8ff
📒 Files selected for processing (8)
src/aiida_agents/agents/analysis/prompt.mdsrc/aiida_agents/cli/agent.pysrc/aiida_agents/cli/config.pysrc/aiida_agents/cli/doctor.pysrc/aiida_agents/cli/mcp.pysrc/aiida_agents/cli/ollama.pysrc/aiida_agents/cli/output.pytests/cli/test_output.py
🚧 Files skipped from review as they are similar to previous changes (5)
- src/aiida_agents/cli/mcp.py
- tests/cli/test_output.py
- src/aiida_agents/cli/agent.py
- src/aiida_agents/cli/output.py
- src/aiida_agents/cli/config.py
- `rag search --refs-only`: print just each match's `source § section` (one per line), for scanning or piping instead of the full panels. - `rag clear` takes `-f/--force` (was `--yes`) to skip its prompt, matching verdi and `rag build`/`config init`; the unskippable prompts (Ollama pull, docs-toolchain install, HITL approval) keep no flag. - split `rag_build` into a 3-line manifest over `_build_with_progress` (the live progress bar) and `_report_build_outcome`, which is now a tested pure function rather than buried in the pragma'd command. - flatten `_tool_parts` from a nested `for/for/if` generator into a single generator expression (same lazy iterator).
A second newline key alongside Esc+Enter, friendlier in vi mode where Esc is the mode switch. Ctrl+J sends `\n` (`ControlJ`), distinct from Enter's `\r` (`ControlM`), and most terminals (Windows Terminal included) emit that same `\n` for Ctrl+Enter, so Ctrl+Enter works too. Shift+Enter can't be bound: terminals don't send a distinct byte for it, so there's no key for prompt_toolkit to catch.
`_run_repl` was one ~65-line function mixing input handling, command dispatch, and query execution. Extract two helpers, leaving the loop as a clean read -> dispatch (quit/`/clear`) -> run-turn skeleton: - `_make_session(repl_cfg)` builds the PromptSession (history + key bindings). - `_run_turn(agent, question, history, repl_cfg) -> history` runs one query, renders the reply, and returns the updated history (unchanged on interrupt/error, so a failed turn never corrupts the conversation). Behaviour-preserving (still `# pragma: no cover`; the interactive loop is verified by dogfooding). The banner also now advertises Ctrl+Enter. Positions for a future `ChatSession` class once more slash-commands land, at which point these become methods and history/session become instance state.
Summary by CodeRabbit
aiida-agentsCLI with chat,ask,check,warm,doctor,config(show/path/init),mcp serve, plusrag(build/status/search/clear) and Ollama on-demand setup.