Skip to content

Integration: Replace all remaining Chainlit usage with PraisonAIUI (aiui) and wire native DB/sessions #1443

@MervinPraison

Description

@MervinPraison

Overview

Replace all remaining Chainlit usage in PraisonAI with the in-house PraisonAIUI (aiui) package, continuing the work already started for praisonai ui and praisonai claw. Chainlit is a heavy (150+ MB transitive, pinned >=2.8.5,<=2.9.4), third-party dependency with its own DB schema and lifecycle that duplicates functionality PraisonAI already has natively (SessionStoreProtocol, HookRegistry, StreamEventEmitter, ApprovalRegistry, multi-provider AgentTeam). aiui is already our sanctioned browser UI and is wired in via praisonai[ui] / praisonai[claw]. This issue extends the same treatment to every remaining Chainlit launch point (--ui chainlit, realtime, bot UI, chat UI, code UI, colab, chainlit_ui.py, chat/app.py) and wires PraisonAI's native DB/sessions directly into aiui so the dashboard persists out of the box.

Background

Why now

  • praisonai ui and praisonai claw already delegate to aiui via subprocess.run(["aiui", "run", app.py, ...]) — see @/Users/praison/praisonai-package/src/praisonai/praisonai/cli/commands/ui.py:86-104 and @/Users/praison/praisonai-package/src/praisonai/praisonai/cli/commands/claw.py:83-103.
  • Five pyproject extras still pull Chainlit: chat, code, realtime, all, claw — see @/Users/praison/praisonai-package/src/praisonai/pyproject.toml:50,60,82,123,198.
  • Chainlit-specific shims already exist because the upstream API keeps moving: @/Users/praison/praisonai-package/src/praisonai/praisonai/ui/chainlit_compat.py:1-246 (patches 3 separate breaking changes across Chainlit versions). This is a maintenance burden we own but get no value from.
  • aiui has protocol-driven providers (BaseProvider + RunEvent), pluggable BaseDataStore, and a SessionProtocol that already knows how to delegate to praisonaiagents.session.SessionStoreProtocol — see @/Users/praison/praisonaiui/src/praisonaiui/provider.py:31-93, @/Users/praison/praisonaiui/src/praisonaiui/datastore.py:28-85, @/Users/praison/praisonaiui/src/praisonaiui/features/sessions_ext.py:1-80.

Design goals

  • Zero Chainlit imports in praisonai after this change. The chainlit dependency is removed from all extras; praisonai[ui], praisonai[chat], praisonai[code], praisonai[realtime], praisonai[claw], praisonai[all] all pin aiui instead.
  • All Chainlit-backed CLI commands keep working: praisonai chat, praisonai code, praisonai realtime, praisonai --ui chainlit (deprecated → aiui), praisonai bot --ui (if it currently opens Chainlit).
  • Native PraisonAI persistence — sessions, memory, approvals, feedbacks stored via praisonaiagents.session.SessionStoreProtocol / praisonai.hooks instead of Chainlit's SQLAlchemy threads/steps/elements schema. Database backend is still pluggable (SQLite default, Postgres opt-in), consistent with how the rest of PraisonAI already works.
  • Agent-centric: the UI is a thin shell over a Core SDK Agent/AgentTeam; no duplicated state.
  • No perf impact: aiui is already lazily loaded via subprocess.run in the UI commands. Same pattern applies to the new replacements.

Architecture Analysis

Current Chainlit surface in PraisonAI

File LOC Purpose Launched by
praisonai/ui/chat.py 889 Full chat UI (tools, memory, crawl, tavily) create_chainlit_chat_interface() in main.py:5183
praisonai/ui/code.py 789 Code assistant UI with ACP/LSP tools create_code_interface() in main.py:5198
praisonai/ui/agents.py 877 Agents YAML runner UI create_chainlit_interface() in main.py:5254, used by --ui chainlit (main.py:728)
praisonai/ui/realtime.py 493 OpenAI Realtime voice (RealtimeClient) create_realtime_interface() in main.py:5275, realtime CLI and flag (main.py:577, 1231)
praisonai/ui/bot.py 426 Bot UI with Steps/streaming praisonai ui bot subcommand
praisonai/ui/chainlit_compat.py 246 Cross-version Chainlit shim (storage/data layer) imported by db.py, chat.py, others
praisonai/ui/db.py 294 DatabaseManager(SQLAlchemyDataLayer) — creates Chainlit threads/steps/elements schema all ui/*.py
praisonai/ui/sql_alchemy.py ~? Chainlit SQLAlchemyDataLayer subclass db.py
praisonai/ui/colab_chainlit.py 80 Colab launcher for Chainlit colab.py
praisonai/ui/colab.py 473 Colab boot script praisonai-colab entry
praisonai/chainlit_ui.py 304 Legacy chainlit app (separate from ui/) Historical entrypoint
praisonai/chat/app.py 125 Legacy chainlit chat app Historical entrypoint
praisonai/cli/main.py 6806 CHAINLIT_AVAILABLE, _get_chainlit_run, create_*_interface CLI entry

How praisonai ui (already done) works today

praisonai ui
  └── cli/commands/ui.py:ui()
        └── subprocess.run(["aiui", "run", ~/.praisonai/ui/app.py, ...])
              └── aiui CLI boots uvicorn + BaseProvider (PraisonAI)
                    └── user's app.py: set_style + set_pages + register_agent

The default app.py is bundled at @/Users/praison/praisonai-package/src/praisonai/praisonai/ui_chat/default_app.py and copied to ~/.praisonai/ui/app.py on first run. Same pattern for claw at @/Users/praison/praisonai-package/src/praisonai/praisonai/claw/default_app.py.

How aiui already integrates with PraisonAI

  • Provider: @/Users/praison/praisonaiui/src/praisonaiui/providers/ ships a PraisonAI provider that bridges StreamEventEmitterRunEvent (token streaming, tool-call events, reasoning steps).
  • Sessions: @/Users/praison/praisonaiui/src/praisonaiui/features/sessions_ext.py defines SessionProtocol with 5 methods (get_session, get_chat_history, add_message, clear_session, delete_session) — structurally identical to praisonaiagents.session.SessionStoreProtocol. Already designed to swap in the SDK store.
  • DataStore: @/Users/praison/praisonaiui/src/praisonaiui/datastore.py provides BaseDataStore (async, 6 methods). Ships MemoryDataStore and JSONFileDataStore. Subclass for SQLite/Postgres/Redis/Mongo — see module docstring lines 16-25.
  • Memory, TTS, Knowledge, Approvals, Sessions: all shipped as aiui features under @/Users/praison/praisonaiui/src/praisonaiui/features/.
  • Gateway bridge: @/Users/praison/praisonaiui/src/praisonaiui/integration.py:35 (AIUIGateway) already exists for WebSocket integration with praisonai.gateway.WebSocketGateway.

Key File Locations

File Purpose Lines
@/Users/praison/praisonai-package/src/praisonai/praisonai/cli/commands/ui.py Precedent: praisonai ui → aiui 107
@/Users/praison/praisonai-package/src/praisonai/praisonai/cli/commands/claw.py Precedent: praisonai claw → aiui 104
@/Users/praison/praisonai-package/src/praisonai/praisonai/ui_chat/default_app.py Default clean-chat aiui app.py ~60
@/Users/praison/praisonai-package/src/praisonai/praisonai/claw/default_app.py Default dashboard aiui app.py ~110
@/Users/praison/praisonai-package/src/praisonai/praisonai/cli/main.py:144-170 CHAINLIT_AVAILABLE + _get_chainlit_run() 27
@/Users/praison/praisonai-package/src/praisonai/praisonai/cli/main.py:5183-5297 4 create_*_interface methods 115
@/Users/praison/praisonai-package/src/praisonai/praisonai/ui/chainlit_compat.py Version shim for Chainlit 246
@/Users/praison/praisonaiui/src/praisonaiui/datastore.py Pluggable DataStore protocol 294
@/Users/praison/praisonaiui/src/praisonaiui/features/sessions_ext.py Session protocol (delegates to SDK) ~300
@/Users/praison/praisonaiui/src/praisonaiui/provider.py Provider protocol + RunEvent 94
@/Users/praison/praisonai-package/src/praisonai-agents/praisonaiagents/session/protocols.py Core SDK SessionStoreProtocol ~100

Gap Analysis Summary

Critical gaps (blockers — must fix before merging)

Gap Impact Severity Placement Effort
praisonai realtime uses Chainlit cl.on_audio_chunk + RealtimeClient Voice UI breaks without aiui replacement Critical aiui (new feature) Large
praisonai --ui chainlit launches ui/agents.py (Chainlit) Need aiui dashboard for YAML agents High praisonai (wrapper) Medium
praisonai ui bot launches ui/bot.py (Chainlit) with Step UI Bot UI breaks High praisonai + aiui Medium
chainlit still in 5 extras in pyproject.toml Users still pull 150+ MB Chainlit Critical praisonai/pyproject.toml Small
ui/chainlit_compat.py, ui/db.py, ui/sql_alchemy.py deletion 836 LOC dead after switch High praisonai Small (delete)

Feature gaps (PraisonAIUI feature requests — file in MervinPraison/PraisonAIUI)

Chainlit Feature aiui Current Support Gap PraisonAIUI Issue
cl.on_audio_chunk + OpenAI Realtime WebRTC features/tts.py one-way TTS only Bidirectional voice realtime Feature request: WebRTC voice realtime
cl.Step (nested chain-of-thought with input/output display) RunEventType.REASONING_STEP exists Step UI component with nested sub-steps and collapsible input/output Feature request: Step / CoT UI
cl.Image, cl.Pdf, cl.File, cl.Video, cl.Audio (inline elements) Message.elements field exists, renderer unclear Rich inline element rendering (image viewer, PDF viewer, file download) Feature request: Message elements rendering
cl.feedback (thumbs up/down per message, persisted) Feedback API + DataStore table Feature request: Message feedback
cl.input_widget (TextInput/Switch/Select in chat settings) set_settings() exists Verify schema + runtime update flow parity Feature request: Chat settings widgets parity
cl.ChatProfile (profiles with per-profile settings + icon) @aiui.profiles exists Verify per-profile settings/starters parity Feature request: Profiles parity
Chainlit SQLAlchemyDataLayer (Postgres/SQLite auto schema) MemoryDataStore, JSONFileDataStore only SQLAlchemyDataStore ships in aiui (SQLite + Postgres) Feature request: Built-in SQL datastore
Chainlit @cl.password_auth_callback, OAuth aiui.auth module Verify OAuth providers parity Feature request (if gap): OAuth parity

DB / Sessions integration

Chainlit's value-add was its bundled DB (threads, steps, elements, feedbacks). PraisonAI already has:

  • Core SDK: SessionStoreProtocol (@/Users/praison/praisonai-package/src/praisonai-agents/praisonaiagents/session/protocols.py), 5-method interface, runtime-checkable. Implementations: in-memory, hierarchical (file-backed), Redis, MongoDB (wrapper side).
  • Wrapper: praisonai has paths and ~/.praisonai/ data directory with SQLite-capable session stores.
  • Memory: praisonaiagents.memory already persists conversation/user/task memory independently of UI.

Plan: ship PraisonAISessionDataStore(BaseDataStore) inside PraisonAIUI (or as a thin adapter in praisonai/ui/ package) that wraps any SessionStoreProtocol implementation. By default aiui running under praisonai uses this adapter, so database support that PraisonAI already has becomes the dashboard's persistence layer automatically — SQLite by default, swap to Postgres/Redis via the same config the Core SDK uses.


Proposed Implementation

Phase 0 — Ship PraisonAI↔aiui datastore adapter (unblocks everything)

Create praisonai/ui/_aiui_datastore.py (new, wrapper side only — zero Core SDK changes):

"""Bridge praisonaiagents SessionStore → aiui BaseDataStore.

Allows any SessionStoreProtocol implementation (file/Redis/Mongo) to back
the aiui dashboard's Sessions page. No Chainlit required.
"""
from __future__ import annotations
from typing import Any, Optional
from praisonaiui.datastore import BaseDataStore
from praisonaiagents.session import (
    SessionStoreProtocol,
    get_hierarchical_session_store,
)


class PraisonAISessionDataStore(BaseDataStore):
    def __init__(self, store: SessionStoreProtocol | None = None):
        self._store = store or get_hierarchical_session_store()

    async def list_sessions(self) -> list[dict[str, Any]]:
        # Iterate via store API
        ...

    async def get_session(self, session_id: str) -> Optional[dict[str, Any]]:
        if not self._store.session_exists(session_id):
            return None
        return {
            "id": session_id,
            "messages": self._store.get_chat_history(session_id),
        }

    async def create_session(self, session_id=None):
        sid = session_id or self._new_id()
        # session_exists() check + lazy create on first add_message
        return {"id": sid, "messages": []}

    async def delete_session(self, session_id: str) -> bool:
        return self._store.delete_session(session_id)

    async def add_message(self, session_id, message):
        self._store.add_message(
            session_id=session_id,
            role=message["role"],
            content=message["content"],
            metadata=message.get("metadata"),
        )

    async def get_messages(self, session_id):
        return self._store.get_chat_history(session_id)

Wire-in inside ui_chat/default_app.py and claw/default_app.py:

import praisonaiui as aiui
from praisonai.ui._aiui_datastore import PraisonAISessionDataStore

aiui.set_datastore(PraisonAISessionDataStore())

Phase 1 — Replace Chainlit launch points with aiui subcommands

Add subcommands to praisonai/cli/commands/ui.py (mirrors memory ac01f186-3eac-4c0a-bfee-92fc809f46b3 — terminal commands stay terminal, browser commands explicitly say ui):

Old launch New launch Default app file
praisonai --ui chainlit (deprecated, ui/agents.py) praisonai ui agents (or keep --ui chainlit as alias that prints deprecation + routes to aiui) praisonai/ui_agents/default_app.py (new)
praisonai realtime / --realtime (ui/realtime.py) praisonai ui realtime praisonai/ui_realtime/default_app.py (new)
praisonai ui bot (ui/bot.py) praisonai ui bot praisonai/ui_bot/default_app.py (new)
create_chainlit_chat_interface / chat/app.py / chainlit_ui.py (dead, not wired) (delete)
create_code_interface (ui/code.py, dead, not wired) (delete)

Every new default_app.py is a thin aiui script: set style, register agents from YAML or kwargs, delegate all streaming/tool-calls to the aiui PraisonAI provider.

Realtime is the hardest case — it needs bidirectional WebRTC audio. Proposal:

  1. Short term: praisonai ui realtime prints "Realtime UI migration pending — see https://github.com/MervinPraison/PraisonAIUI/issues/" and falls back to terminal realtime if available.
  2. Medium term (blocked by PraisonAIUI issue): once aiui ships WebRTC voice, swap in aiui run realtime_app.py.

Phase 2 — Delete Chainlit code & extras

Delete (after grep confirms zero imports):

  • praisonai/ui/chainlit_compat.py (246 L)
  • praisonai/ui/db.py (294 L)
  • praisonai/ui/sql_alchemy.py
  • praisonai/ui/colab_chainlit.py (80 L)
  • praisonai/ui/chat.py (889 L)
  • praisonai/ui/code.py (789 L)
  • praisonai/ui/agents.py (877 L)
  • praisonai/ui/bot.py (426 L)
  • praisonai/ui/realtime.py (493 L) — only after PraisonAIUI realtime lands
  • praisonai/chainlit_ui.py (304 L, legacy)
  • praisonai/chat/app.py (125 L, legacy)
  • praisonai/cli/main.py:144-170 (CHAINLIT_AVAILABLE, _get_chainlit_run)
  • praisonai/cli/main.py:5183-5297 (4 create_*_interface methods)

Update praisonai/pyproject.toml:

 chat = [
-    "chainlit>=2.8.5,<=2.9.4",
-    "aiosqlite>=0.20.0",
-    "greenlet>=3.0.3",
+    "aiui>=0.3.100",
     "tavily-python==0.5.0",
     "crawl4ai>=0.7.0",
-    "sqlalchemy>=2.0.36",
     "playwright>=1.47.0",
     "rich",
 ]

Same diff for code, realtime, claw, all.

Phase 3 — Deprecation warnings

praisonai --ui chainlit prints a one-time warning and silently routes to aiui, to keep existing scripts working one release:

if args.ui == "chainlit":
    import warnings
    warnings.warn(
        "--ui chainlit is deprecated; using aiui. Remove --ui to suppress.",
        DeprecationWarning,
    )
    # fall through to aiui agents launch

Drop the flag entirely in the following minor release.


Files to Create / Modify

New files (wrapper only)

File Purpose
praisonai/ui/_aiui_datastore.py PraisonAISessionDataStore(BaseDataStore) adapter
praisonai/ui_agents/__init__.py
praisonai/ui_agents/default_app.py aiui app for YAML-runner UI (replaces ui/agents.py)
praisonai/ui_bot/__init__.py
praisonai/ui_bot/default_app.py aiui app for Bot UI (replaces ui/bot.py)
praisonai/ui_realtime/__init__.py
praisonai/ui_realtime/default_app.py aiui app for Realtime UI (replaces ui/realtime.py)
tests/unit/test_aiui_datastore.py TDD for adapter (memory + hierarchical backends)
tests/integration/test_ui_subcommands.py Smoke: each praisonai ui <sub> starts without error

Modified files (wrapper only)

File Change
praisonai/cli/commands/ui.py Add agents, bot, realtime subcommands mirroring current ui callback
praisonai/cli/main.py Delete CHAINLIT_AVAILABLE, _get_chainlit_run(), 4 create_*_interface methods, --ui chainlit branch routes to aiui with deprecation warning
praisonai/cli/commands/realtime.py Route to ui realtime subcommand
praisonai/pyproject.toml Drop chainlit from chat, code, realtime, all, claw extras; bump aiui>=0.3.100 minimums
praisonai/ui_chat/default_app.py Add set_datastore(PraisonAISessionDataStore())
praisonai/claw/default_app.py Add set_datastore(PraisonAISessionDataStore())
praisonai/cli/features/doctor/checks/env_checks.py Remove Chainlit check; add aiui check
tests/unit/test_chainlit_compat.py Delete (after chainlit_compat.py deleted)

Deleted files

See Phase 2 above.


Technical Considerations

Dependencies

  • Drop: chainlit>=2.8.5,<=2.9.4 (all 5 extras). Keep sqlalchemy, aiosqlite only where used by praisonai.session (unrelated to UI).
  • Keep/bump: aiui>=0.3.100. Already in [ui] and [claw].

Performance impact

  • aiui already subprocess-launches via aiui run app.py (see cli/commands/ui.py:89); no new import cost on the praisonai CLI hot path.
  • Removing Chainlit's literalai, chainlit, socketio transitive deps saves ~150 MB install size and ~2-3s cold pip install time.
  • PraisonAISessionDataStore is a thin wrapper; lookups are O(1) and only happen on Sessions page load.

Safety / approval

  • No Core SDK changes. All work is in praisonai wrapper + praisonaiui package (external).
  • Backward compatibility: --ui chainlit prints a DeprecationWarning for one release, then removed.
  • Each new default_app.py is copied to ~/.praisonai/ui_<name>/app.py on first run (mirrors ui_chat/claw pattern) so user edits persist across upgrades.

Multi-agent safety

  • aiui provider already bridges AgentTeam (Core SDK class alias for AgentManager) via existing praisonaiui.providers code. No new multi-agent code needed.

Database integration

  • PraisonAISessionDataStore delegates to praisonaiagents.session.SessionStoreProtocol. Whichever backend the Core SDK is configured with (file, Redis, Mongo, custom) becomes the dashboard's persistence layer automatically.
  • Memory, knowledge, and approvals are already surfaced by aiui through its own features module — they reach PraisonAI via the BaseProvider bridge with zero duplication.

Acceptance Criteria

Functional

  • praisonai ui works (already done, must not regress)
  • praisonai claw works (already done, must not regress)
  • praisonai ui agents launches aiui dashboard with YAML-defined agents from agents.yaml
  • praisonai ui bot launches aiui bot UI with streaming + tool-call visualization
  • praisonai ui realtime works OR prints clear message linking to the PraisonAIUI realtime issue until it lands
  • praisonai --ui chainlit prints deprecation warning and routes to praisonai ui agents
  • praisonai realtime routes to praisonai ui realtime
  • Sessions persist across restart via PraisonAISessionDataStore (SQLite default at ~/.praisonai/sessions/)
  • Memory pane in dashboard reflects praisonaiagents.memory writes from a live agent conversation

Non-functional

  • grep -rln "chainlit" praisonai/ --include="*.py" | grep -v tests/ returns zero results
  • grep -n "chainlit" pyproject.toml returns zero results
  • pip install "praisonai[chat]" does not install chainlit
  • python -c "from praisonai.cli.main import PraisonAI" import time < 300 ms (measured before/after)
  • All unit tests pass; test_chainlit_compat.py deleted
  • New test_aiui_datastore.py covers: memory backend, hierarchical backend, session not-found, delete, message ordering

Tests (copy-pastable)

# Unit — adapter
pytest tests/unit/test_aiui_datastore.py -v

# Integration — each subcommand starts
pytest tests/integration/test_ui_subcommands.py -v

# Smoke — no chainlit imports anywhere
! grep -rln "chainlit" src/praisonai/praisonai/ --include="*.py"

# Real agentic — dashboard with live agent
OPENAI_API_KEY=... praisonai ui &
UI_PID=$!
sleep 5
curl -fsS http://localhost:8081/api/health
curl -fsS -X POST http://localhost:8081/api/chat/send \
  -H "Content-Type: application/json" \
  -d '{"message":"Say hello in one sentence","session_id":"smoke"}'
kill $UI_PID

Local verification before merge (explicit user ask)

The user asked for explicit local testing after PR completion:

# 1. Pull PR branch
gh pr checkout <pr-number>

# 2. Fresh install in test venv
python -m venv .venv-test && source .venv-test/bin/activate
pip install -e "src/praisonai[all]"

# 3. Verify no Chainlit pulled
pip list | grep -i chainlit  # should be empty

# 4. Launch each command; visual inspection
praisonai ui &               # browser: http://localhost:8081
praisonai claw &             # browser: http://localhost:8082
praisonai ui agents &        # browser: aiui dashboard with agents.yaml agents
praisonai ui bot &           # browser: aiui bot UI
praisonai --ui chainlit      # must print deprecation, still work

# 5. Session persistence smoke
praisonai ui
# → open browser, send message, Ctrl-C, relaunch → message still in Sessions pane

Implementation Notes

Key Files to Read First

  1. @/Users/praison/praisonai-package/src/praisonai/praisonai/cli/commands/ui.py (107 lines) — the template every new subcommand copies.
  2. @/Users/praison/praisonai-package/src/praisonai/praisonai/ui_chat/default_app.py (~60 lines) — the template every new default_app.py copies.
  3. @/Users/praison/praisonai-package/src/praisonai/praisonai/claw/default_app.py (~110 lines) — dashboard-style app template.
  4. @/Users/praison/praisonaiui/src/praisonaiui/datastore.py (294 lines) — BaseDataStore contract your adapter implements.
  5. @/Users/praison/praisonai-package/src/praisonai-agents/praisonaiagents/session/protocols.py — SessionStoreProtocol contract your adapter delegates to.
  6. @/Users/praison/praisonai-package/src/praisonai/praisonai/cli/main.py:5183-5297 — the 4 create_*_interface methods you are deleting.
  7. @/Users/praison/praisonai-package/src/praisonai/pyproject.toml:50,60,82,123,198 — the 5 extras to clean up.

Critical Integration Points

  1. aiui provider already handles streaming + tools (praisonaiui/providers/). Do not duplicate.
  2. Session adapter is the only bridge code needed for DB persistence. Everything else flows through existing aiui features.
  3. Default app files must stay small and readable — users open them and edit. No hidden behaviour.
  4. praisonai bot (CLI bots for Telegram/Discord/Slack) is unrelated to ui/bot.py — do not touch bot CLI.
  5. ui/colab.py imports ui/colab_chainlit.py — review colab flow and either port to aiui or drop Colab-specific launcher.

PraisonAIUI feature requests (file as separate issues in MervinPraison/PraisonAIUI)

Blockers: realtime voice. High: Step/CoT UI, message feedback, SQL datastore, inline element rendering (image/pdf/file/audio/video). Medium: chat settings widget parity, profile parity.


References

Metadata

Metadata

Assignees

No one assigned

    Labels

    claudeAuto-trigger Claude analysisdocumentationImprovements or additions to documentationenhancementNew feature or requestperformance

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions