Skip to content

feat: web chat channel with SSE token streaming and multi-session UI#1341

Closed
dmagyar wants to merge 26 commits intoHKUDS:mainfrom
dmagyar:feature/web-chat-channel
Closed

feat: web chat channel with SSE token streaming and multi-session UI#1341
dmagyar wants to merge 26 commits intoHKUDS:mainfrom
dmagyar:feature/web-chat-channel

Conversation

@dmagyar
Copy link
Copy Markdown

@dmagyar dmagyar commented Feb 28, 2026

Summary

Adds a self-contained browser-based chat interface to nanobot via a new `WebChannel`.

Core channel (`nanobot/channels/web.py`)

  • HTTP server (aiohttp) with POST `/api/chat` returning a Server-Sent Events stream
  • SSE events: `token` (live LLM text), `progress`, `done`, `error`
  • Per-request asyncio queue bridges the agent loop to the SSE stream
  • Keep-alive pings every 120 s; nginx buffering disabled via `X-Accel-Buffering: no`
  • Integrates with the plugin channel architecture (`discover_all()`) — accepts both `WebConfig` and raw dict config
  • `agent_loop` injected by `ChannelManager` after channel discovery

Agent loop integration (`nanobot/agent/loop.py`)

  • Uses `on_stream` / `on_stream_end` callbacks for token-level streaming
  • `on_tool_result` callback wired through `_LoopHook.after_iteration` to stream tool output back to the browser
  • `process_direct` passes `media` list for image attachment support
  • No changes to other channels — fully backwards compatible
  • Kept in sync with upstream `runner.run(AgentRunSpec(...))` refactor

Server-side session storage

  • Sessions persisted as `~/.nanobot/web-sessions/{uuid}.md`
  • YAML frontmatter + embedded `` block for lossless round-trip
  • Human-readable `### human` / `### assistant` sections for manual inspection
  • REST API: `GET/POST /api/sessions`, `GET/PUT/PATCH/DELETE /api/sessions/{id}`
  • Legacy localStorage data migrated automatically on first load

File & image attachment support

  • `POST /api/upload` — multipart upload saved to `~/.nanobot/web-uploads/`
  • `GET /api/uploads/{id}` — serve uploaded file
  • Images routed as multimodal `image_url` content blocks (vision-capable models)
  • Text files injected as `` blocks in the message
  • Drag-and-drop onto the chat area supported

Frontend (`web/`)

  • Multi-session sidebar — conversations stored server-side, click to switch, first message auto-names the session, pin/rename/delete support
  • SSE token streaming — tokens appear live with a blinking cursor; final text rendered as Markdown via `marked.js`
  • Tool output panels — each tool call rendered in a collapsible panel (click header to expand/collapse); streaming output shown live inside the panel
  • Base64 image rendering — raw base64 image strings detected by magic-byte prefix and rendered as inline `` elements
  • Image lightbox — click any image to open full-screen overlay; close via ✕, backdrop click, or Escape
  • File upload UI — paperclip button, upload preview bar with thumbnails, drag-and-drop
  • Stop button — cancel generation mid-stream
  • Light / dark theme via `prefers-color-scheme`
  • Service worker for offline caching / PWA support
  • Mobile-responsive layout

Config (`nanobot/config/schema.py`)

  • `WebConfig` with `host`, `port`, `static_dir`, `allow_from`, `vision` fields
  • `vision: false` disables sending images to the LLM (for models without vision support)
  • `ChannelsConfig` uses `extra="allow"` — web channel discovered dynamically like all others

Docker (`Dockerfile`)

  • `web/` directory packaged into the wheel via hatchling `force-include`
  • `openssh-client` added for bridge npm install

pyproject.toml

  • Added `[web]` optional dep group: `aiohttp>=3.9.0,<4.0.0`

Enable the web channel

{
  "channels": {
    "web": {
      "enabled": true,
      "host": "0.0.0.0",
      "port": 8080
    }
  }
}
pip install -e ".[web]"
nanobot gateway

Test plan

  • docker compose build && docker compose up — server starts, /api/health returns {"status":"ok"}
  • Open http://localhost:8080 — chat UI loads, empty state shown
  • Send a message — tokens stream live, final answer rendered as Markdown
  • Send a message that triggers tool calls — each tool appears as a collapsible panel with streaming output
  • Upload a text file — contents injected into message context
  • Upload an image (vision-capable model) — image included in LLM request
  • Bot response with base64 image — renders inline, click opens lightbox
  • Sessions persist across page reload
  • Open a second browser tab — sessions are independent, sidebar syncs
  • Drag and drop file onto chat area — attachment added to preview bar
  • Stop button cancels mid-stream generation

@dmahurin
Copy link
Copy Markdown

dmahurin commented Feb 28, 2026

Note, now there are 3 web chat PR's, they all have unique feature/advantages/simplicity

#1047
#1748

https://github.com/search?q=repo%3AHKUDS%2Fnanobot+%22web+chat%22&type=pullrequests

@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Feb 28, 2026

I saw those - and I'm not married to my implementation. I like this UX better. Maybe implementing the backend APIs should be in the project and the associated front-end could be a separate project (so there could be multiple flavours).
Here is an example screenshot from my PR:

image

@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Mar 1, 2026

@dmahurin is there a similar PR that you intend to merge in? I'm okay closing this and use the one you will eventually include. Once the APIs are there (for streaming, tool calls/responses) I can just focus on making an UI as a separate project.

@dmahurin
Copy link
Copy Markdown

dmahurin commented Mar 2, 2026

@dmagyar, I do not have a preference on any specific web chat implementation. This is more a question for @Re-bin on what kind of web chat implementation would be good enough. Minimal, Full featured, or somewhere in between.

@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Mar 8, 2026

Friendly reminder @Re-bin, @dmahurin - should I keep updating this PR in the hopes of a merge or you guys picked one of the other web fronts? Happy to keep this updated if you need more time.

dmagyar and others added 9 commits March 12, 2026 08:29
Adds a new `web` channel that exposes nanobot via a browser-based chat UI.
Includes token-level LLM streaming, tool call hints, multi-session sidebar,
and conversation history persisted in localStorage.

- `nanobot/channels/web.py` — aiohttp HTTP server; GET / serves static files,
  POST /api/chat returns an SSE stream (`token`, `progress`, `done`, `error`
  events). Calls `agent_loop.process_direct()` directly for streaming support.
- `web/index.html` — minimal chat shell with sidebar + message area
- `web/style.css` — dark/light theme via CSS variables, responsive layout
- `web/app.js` — SSE client, multi-session management (localStorage),
  history replay, collapsible tool-call chips
- `web/nanobot_logo.png` — official nanobot logo served as static asset

- `nanobot/config/schema.py` — adds `WebConfig` (enabled, host, port,
  static_dir, allow_from) to `ChannelsConfig`
- `nanobot/channels/manager.py` — registers `WebChannel` when enabled;
  passes `agent_loop` reference to channel constructor
- `nanobot/cli/commands.py` — forwards `agent` to `ChannelManager` so the
  web channel can call the loop directly
- `nanobot/providers/base.py` — adds `chat_stream(on_token)` with a
  non-breaking fallback to blocking `chat()` for providers that don't stream
- `nanobot/providers/litellm_provider.py` — real streaming via
  `acompletion(..., stream=True)`; accumulates tool-call deltas by index
- `nanobot/providers/custom_provider.py` — real streaming via
  `AsyncOpenAI(..., stream=True)`; same delta reassembly pattern
- `nanobot/agent/loop.py` — threads `on_token` callback through
  `process_direct` → `_process_message` → `_run_agent_loop`; switches
  provider call to `chat_stream` so tokens flow during generation
- `pyproject.toml` — adds `aiohttp` optional dep under `[web]` extra;
  `force-include` maps `web/` → `nanobot/web/` in the installed wheel
- `Dockerfile` — installs `.[web]` extra, copies `web/`, exposes port 8080
- `docker-compose.yml` — maps port 8080 for the gateway service

Enable the web channel in `~/.nanobot/config.json`:
```json
"channels": {
  "web": {
    "enabled": true,
    "host": "0.0.0.0",
    "port": 8080
  }
}
```
Then open http://localhost:8080.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add on_tool_call / on_tool_result callbacks to agent loop, passing
  the full untruncated call string and raw arguments dict
- Wire tool_call / tool_result SSE events in WebChannel, forwarding
  tool name, formatted call string, and raw args to the browser
- Rewrite sendMessage in app.js: token stream is split into sections
  around tool blocks — narrative text freezes into rendered-content,
  inline .tool-block is inserted with a spinning indicator, filled
  when tool_result arrives, then streaming resumes in a new section
- Special-case write_file: header shows the file path in accent colour,
  body renders the full file content in a scrollable code block with a
  distinct background (up to 400px) instead of the raw JSON dump
- Add CSS for .tool-block, .tool-header, .tool-spinner, .tool-output,
  .tool-file-content, .tool-filepath, .tool-truncated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chat() already accepted reasoning_effort but chat_stream() was missing
the parameter, causing a TypeError when reasoning_effort is configured.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add manifest.json (standalone display, accent theme colour, SVG icon)
- Add sw.js: cache-first service worker for static assets, network-only
  for /api/ — enables offline shell and PWA install prompt on Android
- Add icon.svg: custom cat-face icon matching brand colours
- index.html: PWA meta tags (apple-mobile-web-app-*, theme-color,
  viewport-fit=cover), hamburger #btn-menu, #sidebar-overlay backdrop
- style.css: #btn-menu hidden on desktop; @media (max-width: 640px)
  makes sidebar a fixed slide-in drawer with backdrop, sets input
  font-size 16px (prevents iOS zoom), adds safe-area-inset-bottom
  padding, tightens touch targets and message spacing
- app.js: openSidebar/closeSidebar helpers wired to hamburger and
  overlay; sidebar auto-closes on session switch; SW registration

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Backend: POST /api/stop endpoint cancels the active agent asyncio.Task
  for a session; tasks tracked in _agent_tasks dict, cleaned up on finish
- Frontend: send button swaps to a red stop square icon while streaming
  via CSS class toggle (.stopping), reverts to send arrow when idle
- AbortController wired to the fetch so clicking stop immediately closes
  the SSE connection (triggering server-side cancellation via ConnectionResetError)
- Graceful abort handling: partial streamed content is rendered as markdown
  instead of showing an error bubble

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clients with the old v1 cache were serving the pre-stop-button
app.js/style.css. Bumping VERSION evicts the old cache on next load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Hover (desktop) or faintly always-visible (mobile) ⋮ button on each message
- Dropdown shows 📌 Pin/Unpin and 🗑 Delete
- Pinned messages get a left accent border; pin state persists in localStorage
- Delete splices message from history and re-renders with corrected indices
- History push moved before DOM append in sendMessage so user-message index
  is known immediately; bot-message actions attached after the 'done' event
- Global click listener closes any open dropdown when clicking elsewhere
- SW bumped to v3 to bust the cached v2 assets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removed max-width: 1100px and margin: auto from #app so the UI
fills the entire browser window, maximizing conversation space.
SW bumped to v4 to bust the cached stylesheet.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Sidebar is now 260px wide
- Each session item has a ⋮ menu button (visible on hover)
  that opens a dropdown with Pin and Delete actions
- Pinned sessions sort to the top of the session list
- Pinned sessions show a 📌 icon prefix in their title
- Deleting the active session switches to the next available one
- SW bumped to v5 to bust cached assets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dmagyar dmagyar force-pushed the feature/web-chat-channel branch from 0ae4f8d to d5d7306 Compare March 12, 2026 07:29
dmagyar and others added 3 commits March 12, 2026 08:53
- Service worker (v6): navigation requests are network-first so the
  auth proxy can intercept expired sessions.  When any fetch response
  is detected as a cross-origin redirect (auth login page), all window
  clients receive an AUTH_REDIRECT message.
- app.js: listens for AUTH_REDIRECT from the SW and calls
  window.location.reload() so the user sees the login page immediately.
- web.py: CORS middleware adds Access-Control-Allow-Origin: * to every
  response; OPTIONS pre-flight requests are handled explicitly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace localStorage session/history persistence with a REST API
backed by ~/.nanobot/web-sessions/{uuid}.md files.

- web.py: fix start() bug (runner setup was unreachable inside
  _handle_options); add GET/POST /api/sessions and
  GET/PUT/PATCH/DELETE /api/sessions/{id} endpoints; sessions stored
  as markdown with YAML frontmatter + ### human/assistant sections
- app.js: replace all localStorage ops with API calls; in-memory
  session/history cache keeps UI synchronous; migrateLegacyData()
  moves existing localStorage sessions to the API on first load
- sw.js: bump VERSION to v7 for cache busting

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds end-to-end file/image upload capability to the web chat UI.

Backend:
- loop.py: add `media` param to process_direct() → passed to
  InboundMessage so existing multimodal pipeline handles images
- web.py: POST /api/upload (multipart → saves to ~/.nanobot/web-uploads/),
  GET /api/uploads/{id} (serve uploaded file); _handle_chat accepts
  `attachments` array — images routed as media paths (base64 to LLM),
  text files injected as <file> blocks in the message
- web.py: fix session .md format to embed <!--JSON block for lossless
  history reload (preserves hints, attachments, message pinning)

Frontend:
- index.html: paperclip button with hidden file input, upload preview
  bar above the textarea, drag-overlay div in messages area
- app.js: addFiles() upload pipeline with optimistic preview thumbnails,
  drag-and-drop on #main, attachments included in /api/chat payload and
  persisted in session history; appendUserMessage renders inline images
  and file pills; replayHistory restores attachment thumbnails
- style.css: upload preview bar, attachment thumbnails, msg-image /
  msg-file in message bubbles, drag-and-drop dashed overlay
- sw.js: bump VERSION to v8

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dmagyar and others added 9 commits March 12, 2026 19:17
Adds automatic base64 image detection and rendering in the chat UI:

- unwrapBase64Images(): detects raw base64 strings by magic-byte prefix
  (PNG iVBORw0, JPEG /9j/, GIF R0lGOD, WebP UklGR) and bare data: URIs,
  wraps them in markdown image syntax before passing to marked.parse()
- renderMarkdown(): replaces all direct marked.parse() calls; covers
  streamed token buffers, done events, replayed history, and aborted turns
- Tool result display: if output is a raw base64 image, renders <img>
  instead of text (covers code-execution tools returning plot images)
- style.css: .rendered-content img / .tool-result-image — max 480px,
  rounded, zoom-in cursor; applies to all images in bot bubbles
- sw.js: bump VERSION to v9

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clicking any image in the chat (user attachments, bot markdown images,
tool result images) opens a full-screen lightbox:
- Fade + scale-in animation, dark backdrop
- Fixed ✕ close button top-right with glass effect
- Click backdrop or press Escape to close
- Event delegation on #messages — no per-image handlers needed
- body overflow locked while open to prevent scroll bleed
- sw.js: bump VERSION to v10

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Resolve conflicts in loop.py, schema.py, pyproject.toml
- WebChannel now accepts dict config (ChannelsConfig uses extra="allow")
- manager.py injects agent_loop into web channel after discovery
- Keep WebConfig with vision flag; ChannelsConfig now fully dynamic
- Keep web + langsmith as separate optional extras in pyproject.toml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…mpatibility

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ry methods)

- Resolve conflicts in loop.py, custom_provider.py, litellm_provider.py
- Adopt main's on_stream/on_stream_end API; remove on_tool_call/on_tool_result
- Use chat_stream_with_retry / chat_with_retry in agent loop
- process_direct now returns OutboundMessage | None; update web channel accordingly
- Keep media parameter in process_direct for image upload support
- Remove stale tool_call/tool_result SSE events from web channel

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The old feature-branch chat_stream (using on_token) was shadowing the new
one from main (using on_content_delta / max_tokens). Python picks the last
definition, so chat_stream_with_retry was calling the old one without
max_tokens in its signature, causing the unexpected keyword argument error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Mar 23, 2026

Can I please get a verdict on this PR? I'm keeping it updated in the hopes of a merge; also using it in production for quite a while. If you decide not to merge which web channel interface should I use?

dmagyar and others added 3 commits March 23, 2026 16:05
- Normalize dict tool_choice to "required" string in CustomProvider
  (fixes Qwen/llama.cpp peg-native template strict validation)
- Fix consecutive assistant messages when subagent result is injected
  (use user role so strict providers don't reject the context)
- Emit "Thinking..." progress immediately before memory consolidation
  (prevents silent SSE hang while consolidation runs)
- Stream exec stdout line-by-line via on_progress callback
  (real-time output instead of waiting for command to finish)
- Increase default exec timeout from 60s to 120s
- Emit proper tool_call/tool_stream SSE events from web channel
  (tool blocks now render inline instead of floating chips at top)
- Add tool_stream handling in frontend to append exec output live
- Finalize pending tool blocks with checkmark on done event
- Emit tool_result SSE event after each tool call completes so exec
  output always appears even when subprocess buffers stdout
- Reset _tool_active after each tool result so multi-tool turns route
  correctly
- Tool blocks are collapsed by default; auto-expand when output streams
  in (tool_stream) or when result arrives (tool_result)
- Click the tool header to manually toggle expand/collapse
- Add ▶/▼ chevron indicator on tool headers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Apr 5, 2026

Dear maintainers any feedback on this PR? I have not seen a more complete web channel yet. @chengyongru?

@chengyongru chengyongru added the invalid This doesn't seem right label Apr 6, 2026
@dmagyar
Copy link
Copy Markdown
Author

dmagyar commented Apr 6, 2026

Very fair comments @chengyongru - I've submitted a new PR with ONLY the web related features. This will miss some interesting features (like showing tool executions, collapsible thinking logic etc.) but you are right - this should be a clean PR for only adding the web channel. Please find the new PR here: #2871

@dmagyar dmagyar closed this Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

invalid This doesn't seem right

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants