feat: model discoverability & provider auto-selection#407
Merged
Conversation
Design for improving model discovery UX: update stale KNOWN_MODELS, add live model fetching from provider APIs, model selection in onboarding, /model list hints, /model fetch command, and startup mismatch warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
provider_name_for_model() matches model strings against provider keywords. build_runtime_provider_chain() reorders candidates so the keyword-matched provider is promoted to first position.
…aude-sonnet-4-6 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ModelCommand::Fetch variant with parsing, test, stub handler in CLI agent loop, Telegram fallback message, and slash command completion entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add parse_models_openai_format(), parse_models_ollama_format(), and fetch_provider_models() to src/cli/common.rs. Covers Anthropic, OpenAI, Ollama, and all OpenAI-compatible providers. Five unit tests all pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughAdds runtime model discoverability and provider auto-selection: live per-provider model fetching with OpenAI/Ollama parsers and 10s timeout, a CLI Changes
Sequence DiagramsequenceDiagram
participant User as CLI User
participant CLI as CLI Handler
participant Config as Config (providers)
participant Provider as Provider API
participant Parser as Fetch Parsers
User->>CLI: /model fetch
activate CLI
CLI->>Config: read configured providers & creds
Config-->>CLI: provider list
loop for each provider
CLI->>Provider: HTTP GET /models (auth, api_base)
activate Provider
alt success
Provider-->>CLI: JSON models
CLI->>Parser: parse_models_*_format(JSON)
Parser-->>CLI: sorted model IDs
CLI->>User: display provider + models
else failure/timeout
Provider-->>CLI: error
CLI->>User: display provider fetch failure
end
deactivate Provider
end
CLI->>User: hint: Switch: /model <model-id>
deactivate CLI
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/channels/model_switch.rs (1)
188-203:⚠️ Potential issue | 🟠 MajorColon-containing model IDs are still parsed as
provider:model.
split_once(':')turns/model llama3.3:latestinto provider=llama3.3, model=latest. The new/model fetchflow prints Ollama tags in exactly that form and tells users to switch with/model <model-id>, so those live-discovered IDs will not round-trip. Please only treat the left-hand side as a provider when it matches a supported provider id; otherwise keep the whole argument as the model string.Minimal hardening
- if let Some((provider, model)) = arg.split_once(':') { + if let Some((provider, model)) = arg + .split_once(':') + .filter(|(provider, _)| crate::providers::RUNTIME_SUPPORTED_PROVIDERS.contains(provider)) + { Some(ModelCommand::Set(ModelOverride { provider: Some(provider.to_string()), model: model.to_string(), }))🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/channels/model_switch.rs` around lines 188 - 203, The parsing treats any "a:b" as provider:model; change the logic in src/channels/model_switch.rs where you build ModelCommand::Set(ModelOverride) so you only split on ':' when the left-hand token is a supported provider id—e.g., replace the current arg.split_once(':') branch with one that does if let Some((provider, model)) = arg.split_once(':') && is_supported_provider(provider) (or use the existing provider parsing helper / Provider::from_str) then set ModelOverride { provider: Some(provider.to_string()), model: model.to_string() } else set provider: None and model: arg.to_string(); keep references to ModelCommand::Set and ModelOverride to locate the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/superpowers/plans/2026-03-23-model-discoverability.md`:
- Around line 945-977: Update the "Task 8: Final checks — fmt, clippy, full test
suite" checklist to include the required repo documentation sync step: after
Step 5 (Final commit if fmt changed anything) add a Step 6 to update CLAUDE.md
with architecture tree, test counts, module descriptions, and any new CLI flags
and to update AGENTS.md with the current project snapshot; renumber subsequent
steps accordingly (e.g., move "Verify format check" to Step 7) and ensure the
new step is described exactly so readers know to perform these documentation
updates as part of finalization.
In `@docs/superpowers/specs/2026-03-23-model-discoverability-design.md`:
- Line 123: Several fenced code blocks in the document are missing language
identifiers and trigger markdownlint MD040; locate the plain code fences ("```"
opening and closing) and add an appropriate language tag after the opening fence
(for example change ``` to ```text or to a specific language like
```json/```bash where applicable) for each offending block so the opening fence
becomes ```text (or the correct language) while keeping the closing fence as
```, ensuring all occurrences flagged (the plain triple-backtick blocks) are
updated consistently.
In `@src/cli/agent.rs`:
- Around line 471-503: The ModelCommand::Fetch branch in the match currently
calls super::common::fetch_provider_models(s.name, &s.api_key, api_base) but
ignores the other resolved RuntimeProviderSelection fields (credential,
auth_header, api_version), which breaks OAuth and custom-header providers;
update the call in the ModelCommand::Fetch arm to pass the full resolved
selection (e.g., &s.credential, &s.auth_header, s.api_version.as_deref(), plus
api_base and name) and then update the fetch_provider_models function signature
in src/cli/common.rs to accept these additional parameters so live discovery
uses the same auth/header/version resolution as runtime.
In `@src/cli/common.rs`:
- Around line 760-762: The current early return uses
zeptoclaw::providers::provider_name_for_model(model).is_some(), which suppresses
the mismatch warning for any known provider even when it's not configured;
change the logic to first get the inferred provider via
zeptoclaw::providers::provider_name_for_model(model) and only suppress the
warning when that Option is Some(provider) AND
configured_providers.contains(provider) (or equivalent membership check on
configured_providers); update the block around this check (the code that
currently returns None) to perform the lookup and containment test instead of a
plain is_some().
- Around line 692-706: The fetch_provider_models function currently falls back
to "https://api.openai.com/v1" for unknown providers causing API keys to be sent
to OpenAI; update it to resolve the provider base URL via the PROVIDER_REGISTRY
(use the registry lookup for the given provider name and fall back only to a
safe default like an empty/none or explicit error), build the models URL from
that resolved base (instead of api_base.unwrap_or("https://api.openai.com/v1")),
and ensure the request header logic (the req variable, provider == "anthropic"
branch, and Authorization header setting) uses the resolved base/provider info
so keys are only sent to the correct provider endpoint; reference
fetch_provider_models, PROVIDER_REGISTRY, provider, api_base, url, and req when
making these changes.
In `@src/cli/onboard.rs`:
- Around line 207-216: The prompt "Choice [1]" currently treats empty input ("")
as "keep existing model" instead of selecting option 1; update the match
handling for input in the onboarding flow (the code around read_line(),
input.trim(), and the match arm that checks "s" | "S" | "") so that "" (empty
string) is interpreted as selecting option 1. Concretely, change the branch so
that when input == "" you assign or set the chosen model to the first option
(e.g., set config.agents.defaults.model to the model for choice 1 or call the
same selection logic used for "1"), instead of printing "Keeping default model";
keep handling for "s"/"S" if that means skipping. Ensure you reference and
update the code that prints/uses config.agents.defaults.model so Enter truly
picks choice 1.
In `@src/config/types.rs`:
- Line 773: Update all places that still hardcode the old default model string
to use the new unified default "claude-sonnet-4-6": change occurrences in the
Claude provider initializations (search for "claude-sonnet-4-5-20250929" in
src/providers/claude.rs), the onboarding defaults in src/cli/onboard.rs, the
cost lookup keys in src/utils/cost.rs (and any mapping or enum that references
the older model), and update any related integration fixtures or test data that
reference the old model identifier so cost lookups and behavior are consistent
with the new default; ensure any switch/case, mapping, or enum names that expect
the older model are adjusted to include "claude-sonnet-4-6" and run tests to
verify cost lookup and onboarding paths use the new default.
In `@src/providers/registry.rs`:
- Around line 277-293: The provider inference in provider_name_for_model
currently uses substring contains over spec.model_keywords which causes wrong
routing (e.g., "anthropic/claude-..." or "llama-..."). Update
provider_name_for_model to first check an exact/prefix mapping or
provider-specific matchers (e.g., exact model map and prefix checks like
"anthropic/" or "provider_name/" patterns) before falling back to the generic
spec.model_keywords heuristic; keep the runtime_supported filter and then map to
spec.name as before. Modify PROVIDER_REGISTRY entries or add a dedicated lookup
(prefix_map or matcher functions) so provider-specific ID formats are matched
deterministically prior to the existing contains-based fallback.
---
Outside diff comments:
In `@src/channels/model_switch.rs`:
- Around line 188-203: The parsing treats any "a:b" as provider:model; change
the logic in src/channels/model_switch.rs where you build
ModelCommand::Set(ModelOverride) so you only split on ':' when the left-hand
token is a supported provider id—e.g., replace the current arg.split_once(':')
branch with one that does if let Some((provider, model)) = arg.split_once(':')
&& is_supported_provider(provider) (or use the existing provider parsing helper
/ Provider::from_str) then set ModelOverride { provider:
Some(provider.to_string()), model: model.to_string() } else set provider: None
and model: arg.to_string(); keep references to ModelCommand::Set and
ModelOverride to locate the change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cff9642e-97b8-41c5-bab0-ffe025924b8e
📒 Files selected for processing (13)
docs/superpowers/plans/2026-03-23-model-discoverability.mddocs/superpowers/specs/2026-03-23-model-discoverability-design.mdsrc/channels/model_switch.rssrc/channels/telegram.rssrc/cli/agent.rssrc/cli/common.rssrc/cli/onboard.rssrc/cli/slash.rssrc/config/mod.rssrc/config/types.rssrc/kernel/provider.rssrc/providers/mod.rssrc/providers/registry.rs
| const COMPILE_TIME_DEFAULT_MODEL: &str = match option_env!("ZEPTOCLAW_DEFAULT_MODEL") { | ||
| Some(v) => v, | ||
| None => "claude-sonnet-4-5-20250929", | ||
| None => "claude-sonnet-4-6", |
There was a problem hiding this comment.
Unify the default model across all hardcoded Claude/default paths
Line 773 updates the config fallback, but other code paths still hardcode claude-sonnet-4-5-20250929 (src/providers/claude.rs, src/cli/onboard.rs, src/utils/cost.rs, and related integration fixtures). This creates inconsistent defaults depending on entry path and can break cost lookups for the new default model.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/config/types.rs` at line 773, Update all places that still hardcode the
old default model string to use the new unified default "claude-sonnet-4-6":
change occurrences in the Claude provider initializations (search for
"claude-sonnet-4-5-20250929" in src/providers/claude.rs), the onboarding
defaults in src/cli/onboard.rs, the cost lookup keys in src/utils/cost.rs (and
any mapping or enum that references the older model), and update any related
integration fixtures or test data that reference the old model identifier so
cost lookups and behavior are consistent with the new default; ensure any
switch/case, mapping, or enum names that expect the older model are adjusted to
include "claude-sonnet-4-6" and run tests to verify cost lookup and onboarding
paths use the new default.
…ing, onboard UX - fetch_provider_models: use PROVIDER_REGISTRY default_base_url instead of hardcoded OpenAI fallback that would leak API keys to wrong endpoint - model_provider_mismatch_warning: check inferred provider is actually configured before suppressing warning; give specific message when provider is known but not configured - parse_model_command/parse_override_value: only split on ':' when left side is a known provider name, preserving Ollama tags like llama3.3:latest - onboarding: empty input now selects choice 1 (matching "[1]" prompt hint) instead of skipping Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/cli/onboard.rs`:
- Around line 177-217: The prompt's Enter behavior always picks models[0], which
can be a different model than the current configured default because
live/fallback ordering differs; change the Enter-case logic so it preserves the
existing default if it's present in the computed models list
(config.agents.defaults.model) and only falls back to models[0] if the current
default isn't in models; update the match "" arm where
config.agents.defaults.model is set (and any related messaging) to look up the
current default in the models Vec returned by fetch_provider_models /
KNOWN_MODELS before assigning, using function/variable names like
fetch_provider_models, KNOWN_MODELS, models, and config.agents.defaults.model to
locate the code to modify.
- Around line 154-168: The selected default provider stored in the local
variable primary is discarded because code later writes only the raw model ID
into config.agents.defaults.model, losing the provider choice (and breaking
disambiguation for overlapping IDs and KNOWN_MODELS entries); change the code
paths that currently persist just the model string (where
config.agents.defaults.model is set) to persist a combined provider+model
identity (e.g., "provider/model" or a small struct/object) when providers.len()
> 1, and update any places that read defaults to parse/consume that combined
value; specifically, modify the logic around primary, the branches that set
config.agents.defaults.model, and any KNOWN_MODELS fallback handling so the
provider from primary is stored and later used at runtime.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 37588e2a-ca97-424f-a76d-f93303111c10
📒 Files selected for processing (3)
src/channels/model_switch.rssrc/cli/common.rssrc/cli/onboard.rs
Comment on lines
+154
to
+168
| // If multiple providers, ask which is primary | ||
| let primary = if providers.len() > 1 { | ||
| println!("Multiple providers configured. Which should be your default?"); | ||
| for (i, p) in providers.iter().enumerate() { | ||
| println!(" {}. {}", i + 1, p); | ||
| } | ||
| println!(); | ||
| print!("Choice [1]: "); | ||
| io::stdout().flush()?; | ||
| let input = read_line()?; | ||
| let idx = input.trim().parse::<usize>().unwrap_or(1).saturating_sub(1); | ||
| providers.get(idx).copied().unwrap_or(providers[0]) | ||
| } else { | ||
| providers[0] | ||
| }; |
There was a problem hiding this comment.
The selected default provider is thrown away here.
primary only decides which catalog to read; every branch later persists just the raw model ID into config.agents.defaults.model. That means overlapping IDs—and the new OpenRouter-style anthropic/... / google/... fallback entries from KNOWN_MODELS—lose the user's provider choice before runtime sees them. Persist provider + model together when multiple providers are configured.
Also applies to: 177-195, 213-245
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/cli/onboard.rs` around lines 154 - 168, The selected default provider
stored in the local variable primary is discarded because code later writes only
the raw model ID into config.agents.defaults.model, losing the provider choice
(and breaking disambiguation for overlapping IDs and KNOWN_MODELS entries);
change the code paths that currently persist just the model string (where
config.agents.defaults.model is set) to persist a combined provider+model
identity (e.g., "provider/model" or a small struct/object) when providers.len()
> 1, and update any places that read defaults to parse/consume that combined
value; specifically, modify the logic around primary, the branches that set
config.agents.defaults.model, and any KNOWN_MODELS fallback handling so the
provider from primary is stored and later used at runtime.
Comment on lines
+177
to
+217
| let models: Vec<String> = if let Some(s) = selection { | ||
| match super::common::fetch_provider_models(s.name, &s.api_key, s.api_base.as_deref()).await | ||
| { | ||
| Ok(m) if !m.is_empty() => m, | ||
| _ => { | ||
| println!(" Could not fetch live models, showing known models."); | ||
| KNOWN_MODELS | ||
| .iter() | ||
| .filter(|km| km.provider == primary) | ||
| .map(|km| km.model.to_string()) | ||
| .collect() | ||
| } | ||
| } | ||
| } else { | ||
| KNOWN_MODELS | ||
| .iter() | ||
| .filter(|km| km.provider == primary) | ||
| .map(|km| km.model.to_string()) | ||
| .collect() | ||
| }; | ||
|
|
||
| if models.is_empty() { | ||
| println!(" No models found for {}. Keeping default.", primary); | ||
| return Ok(()); | ||
| } | ||
|
|
||
| println!(); | ||
| println!("Which model would you like to use?"); | ||
| print!("{}", format_model_menu(&models, 15)); | ||
| println!(); | ||
| print!("Choice [1]: "); | ||
| io::stdout().flush()?; | ||
|
|
||
| let input = read_line()?; | ||
| let input = input.trim(); | ||
|
|
||
| match input { | ||
| "" => { | ||
| // Enter = pick choice 1 (matches the "[1]" prompt hint) | ||
| config.agents.defaults.model = models[0].clone(); | ||
| println!(" Set model to: {}", models[0]); |
There was a problem hiding this comment.
Choice [1] currently depends on catalog ordering, not the intended default.
Live results are sorted in fetch_provider_models(), fallback results keep raw KNOWN_MODELS order, and Enter always selects models[0]. For providers like Anthropic, a blind Enter can land on a completely different default than the one this PR is trying to standardize. Move the intended default to index 0, or make Enter keep the current default.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/cli/onboard.rs` around lines 177 - 217, The prompt's Enter behavior
always picks models[0], which can be a different model than the current
configured default because live/fallback ordering differs; change the Enter-case
logic so it preserves the existing default if it's present in the computed
models list (config.agents.defaults.model) and only falls back to models[0] if
the current default isn't in models; update the match "" arm where
config.agents.defaults.model is set (and any related messaging) to look up the
current default in the models Vec returned by fetch_provider_models /
KNOWN_MODELS before assigning, using function/variable names like
fetch_provider_models, KNOWN_MODELS, models, and config.agents.defaults.model to
locate the code to modify.
- fetch_provider_models: accept full RuntimeProviderSelection to support OAuth credentials, custom auth headers, and api_version (was only using api_key). Updated both callers (agent.rs, onboard.rs). - provider_name_for_model: handle "provider/model" prefix IDs (OpenRouter style) by checking if prefix is a known provider. Remove model-family keywords (llama, mistral, phi, qwen) from Ollama since they're not provider-specific. - Unify default model to claude-sonnet-4-6 in ClaudeProvider, onboarding hardcoded strings, and add to cost pricing table. - Add language tags to fenced code blocks in spec doc (MD040). - Add CLAUDE.md/AGENTS.md sync step to plan doc finalization checklist. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep both test additions (KNOWN_MODELS integrity tests + usage hints test). Keep our language-tagged docs versions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
taqtiqa-mark
pushed a commit
to taqtiqa-mark/zeptoclaw
that referenced
this pull request
Mar 25, 2026
## Summary - **Model-driven provider auto-selection**: setting `model: "gpt-5.4"` now automatically selects OpenAI as provider, even with multiple API keys configured - **Updated KNOWN_MODELS**: current models (GPT-5.4 family, Claude 4.6, Gemini 3.1, etc.) replacing stale entries - **Default model**: `claude-sonnet-4-6` (was `claude-sonnet-4-5-20250929`) - **`/model list` usage hints**: shows how to switch, reset, and configure at the bottom - **`/model fetch`**: live model discovery from provider APIs via `/v1/models` endpoints - **Onboarding model step**: asks which model after API key setup (both express and full paths) - **Startup mismatch warning**: warns when configured model doesn't match any provider keywords ## Test plan - [x] 3270 lib tests pass (includes 10 new tests for provider auto-selection, 3 for KNOWN_MODELS integrity, model list hints, fetch parsing) - [x] 128 doc tests pass - [x] `cargo clippy -- -D warnings` clean - [x] `cargo fmt -- --check` clean - [ ] Manual: `zeptoclaw onboard` shows model selection after provider setup - [ ] Manual: `/model fetch` returns live models from configured providers - [ ] Manual: `/model list` shows usage hints at bottom **Spec:** `docs/superpowers/specs/2026-03-23-model-discoverability-design.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `/model fetch` (CLI) to retrieve live model lists from configured providers * Interactive model-selection step added to onboarding * Startup warning when configured model doesn't match available providers * Provider auto-selection now prefers providers matching the configured model * **Updates** * Default model updated to claude-sonnet-4-6 * Known-models registry refreshed; `/model list` shows usage hints * CLI help/completion updated to include `model fetch` * **Documentation** * Added design spec and implementation plan for model discoverability * **Tests** * New and updated unit tests for parsing, fetching, onboarding, and provider inference <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
model: "gpt-5.4"now automatically selects OpenAI as provider, even with multiple API keys configuredclaude-sonnet-4-6(wasclaude-sonnet-4-5-20250929)/model listusage hints: shows how to switch, reset, and configure at the bottom/model fetch: live model discovery from provider APIs via/v1/modelsendpointsTest plan
cargo clippy -- -D warningscleancargo fmt -- --checkcleanzeptoclaw onboardshows model selection after provider setup/model fetchreturns live models from configured providers/model listshows usage hints at bottomSpec:
docs/superpowers/specs/2026-03-23-model-discoverability-design.md🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
/model fetch(CLI) to retrieve live model lists from configured providersUpdates
/model listshows usage hintsmodel fetchDocumentation
Tests