Skip to content

feat(tools): add web_search and web_fetch agent tools#12

Merged
penso merged 20 commits intomainfrom
web-browsing
Feb 2, 2026
Merged

feat(tools): add web_search and web_fetch agent tools#12
penso merged 20 commits intomainfrom
web-browsing

Conversation

@penso
Copy link
Copy Markdown
Collaborator

@penso penso commented Feb 1, 2026

Summary

  • Add web_search tool (Brave Search / Perplexity Sonar) with in-memory cache, structured JSON results, and setup hints for missing API keys
  • Add web_fetch tool with SSRF protection, manual redirect following with loop detection, HTML→text extraction, and configurable truncation
  • Add SearchProvider enum and web config structs (WebSearchConfig, WebFetchConfig, PerplexityConfig) to config schema
  • Register both tools automatically from config in the gateway

Test plan

  • 33 unit tests covering SSRF blocking, cache hit/miss/expiry, response parsing, config resolution, HTML extraction, truncation, and parameter validation
  • Set BRAVE_API_KEY, start gateway, ask LLM a current-events question → should invoke web_search
  • Ask LLM to summarize a URL → should invoke web_fetch

🤖 Generated with Claude Code

penso and others added 20 commits February 1, 2026 13:44
Adds two new LLM-callable tools: web_search (Brave Search / Perplexity)
and web_fetch (URL fetching with HTML extraction and SSRF protection).
Both tools use in-memory caching with configurable TTL and are registered
automatically from config.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The chat now displays `$ web_fetch <url>` and `$ web_search "<query>"`
instead of just the bare tool name.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When an Apple container exists but is stopped, ensure_ready now calls
`container start` instead of assuming it's running. This fixes exec
errors when the container was created previously but has since stopped.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Apple Container's default entrypoint (/bin/bash) exits immediately
without a TTY, so container exec always fails with "not found". Pass
`sleep infinity` as the init process args — same approach as Docker —
to keep the container alive for exec calls.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add info/debug/warn logs to ensure_ready, exec, and cleanup for the
Apple Container backend. Also add debug logs in the exec tool's sandbox
routing so container creation, restarts, and failures are visible in
the gateway logs.

Also: when container start fails on a stopped container, remove it and
recreate instead of bailing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds containerName to the sandbox info API response and displays it
in the context panel, making it easier to debug sandbox issues.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Debug-level logs were invisible at the default info log level, making
sandbox issues impossible to diagnose. Promote all exec tool routing
decisions and Apple Container lifecycle logs to info.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…xistent containers

`container inspect` returns exit 0 with `[]` for containers that don't
exist, causing ensure_ready to skip creation. Now check the output
content: empty/`[]` triggers creation, `"running"` returns early,
`stopped`/`exited` triggers restart, and unknown states trigger
remove-and-recreate.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Host CWD doesn't exist inside the container, causing `cd` to fail.
Move sandbox detection before working_dir resolution and default to
"/" for sandboxed execution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extract the Accept-Language header from the WebSocket upgrade request,
store it on ConnectedClient, and inject it as _accept_language in the
tool context. web_fetch and web_search (Brave) now send it on outgoing
HTTP requests so fetched pages and search results match the user's
locale.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The ubuntu:25.10 base image is minimal and lacks common tools. After
container creation, run apt-get to install configurable packages
(default: curl, python3, nodejs, npm). The package list is exposed as
`tools.exec.sandbox.packages` in config — set to [] to skip.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add git, jq, wget, build-essential, ripgrep, python3-pip, openssh-client,
zip/unzip, rsync, ca-certificates to the default provisioning list.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…t config

- Extract sandbox_image_tag/sandbox_image_exists as shared helpers
- Implement build_image for AppleContainerSandbox (same Dockerfile approach as Docker)
- Skip apt-get provisioning when container uses a pre-built moltis-sandbox: image
- Remove Docker-only guard on startup pre-build (any backend can now pre-build)
- Remove per-exec provision event emission from exec.rs (no longer needed)
- Add `moltis sandbox list|build|remove|clean` CLI commands
- Handle Apple Container's different CLI (image ls format, image delete vs rm)
- Write default moltis.toml on first run so packages are editable without recompile
- Expand default packages to match GitHub Actions runner images (clang, llvm,
  python3-dev, python3-venv, ruby, libssl-dev, sqlite3, etc.)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Runs in an ubuntu:25.10 container, dumps dpkg -l, extracts the moltis
default packages from schema.rs, and shows the diff. Triggers on
changes to schema.rs or the workflow itself, plus manual dispatch.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Mitigates the same class of vulnerability as GHSA-g8p2-7wf7-98mq in
openclaw: a malicious webpage could connect to the local gateway WebSocket
from the victim's browser, gaining full operator access because loopback
connections are auto-authenticated.

The fix validates the Origin header on WebSocket upgrade requests.
Browsers always send Origin on cross-origin requests; non-browser clients
(CLI, SDKs) typically omit it and are still allowed through. Loopback
variants (localhost, 127.0.0.1, ::1) are treated as equivalent.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- README: add sandboxed execution, WebSocket security, default config,
  and sandbox CLI commands to features list
- CLAUDE.md: add sandbox architecture, security (CSWSH, SSRF), and
  sandbox CLI commands sections
- Add From<&config::SandboxConfig> for tools::SandboxConfig to eliminate
  duplicated field-by-field conversion in server.rs and sandbox_commands.rs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…for web search keys

Add iproute2, net-tools, bison, flex, dpkg-dev, fakeroot, zstd, lz4,
pigz, gnupg2, tzdata, shellcheck, patchelf to default sandbox packages
to match GitHub Actions runner images more closely.

Wrap web search API keys in Secret<String> to prevent accidental logging.
Fix CI package extraction regex to scope to default_sandbox_packages().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
No longer needed — package list verified against ubuntu:25.10 base.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@penso penso merged commit feb97c1 into main Feb 2, 2026
5 checks passed
@penso penso deleted the web-browsing branch February 2, 2026 04:41
penso added a commit that referenced this pull request Mar 23, 2026
feat(tools): add web_search and web_fetch agent tools
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Cstewart-HC added a commit to Cstewart-HC/moltis-mini that referenced this pull request Apr 16, 2026
Review fixes applied:
- #1: Extract require_str/opt_usize_or to moltis_tools::params — replaced
  local helpers with params::require_str() and params::u64_param() from
  the shared workspace crate
- #2: Unified error model — CodebaseSearchTool now returns
  Ok(json!({error:..., search_available: false})) for
  BackendUnavailable, matching Peek/Status pattern
- #3: u64→usize truncating cast replaced with usize::try_from().unwrap_or()
- #4: ensure_collections() error remapped from BackendUnavailable to
  IndexFailed { project_id, message }
- moltis-org#6: result.line as usize now clamped with .max(1) minimum
- moltis-org#7: compute_delta carries forward previous hash on hash errors so
  files aren't spuriously marked as removed
- moltis-org#8: Added doc comment noting that watcher batches may contain
  duplicate paths
- moltis-org#9: Extracted effective_extension() from filter.rs, removed
  duplication between filter.rs and watcher.rs
- moltis-org#11: Added tracing::debug! for skipped files in build_initial_snapshot
- moltis-org#12: Added 'drop to stop' documentation on CodeIndexWatcher::start()

32 tests pass, clippy clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant