fix: Token usage statistics now display via WebSocket log fetching by namastex888 · Pull Request #177 · automagik-dev/genie

namastex888 · 2025-10-21T22:42:40Z

Summary

Fixed token usage statistics display in genie dashboard by implementing WebSocket-based log fetching.

Problem

Token statistics code existed but wasn't displaying during execution
REST API getExecutionProcess() doesn't include logs
Logs only accessible via WebSocket endpoints
Token parser didn't handle WebSocket event format

Solution

Added WebSocket-based log fetching using Forge's /raw-logs/ws endpoint
Extract STDOUT content from JsonPatch operations
Parse newline-delimited JSON events from STDOUT
Added parser for stream_event with message_start containing usage data
Added @types/ws dev dependency for TypeScript support

Changes

.genie/cli/src/lib/token-tracker.ts: WebSocket fetcher + stream_event parser
package.json: Added @types/ws@8.18.1

Testing

✅ Successfully extracts token metrics from completed execution processes
✅ Test metrics: 4,072 input + 2,804 output + 450,125 cached = 6,876 total tokens

Token statistics now display in:

Live dashboard (compact format during genie command)
Shutdown report (detailed format on exit)

- Use gh CLI instead of direct git push to main - Create release branch with version bump - Create GitHub release using gh release create - Create PR to main and auto-merge - Prevent infinite loop by excluding automated release PRs - Add pull-requests: write permission fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

- Allow npm publish to fail if version exists (testing scenario) - Check if release exists before creating (skip if exists) - Check if PR exists before creating (skip if exists) - All operations now idempotent and safe to retry fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

The workflow successfully creates GitHub releases but was failing when trying to add a 'release' label that doesn't exist in the repository. fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Three critical fixes for init/template/LLM configuration: 1. **Fix template parameter in paths.ts** - getTemplateGeniePath() now actually uses the template parameter - Correctly copies from .genie/code/ or .genie/create/ based on choice - Fixes bug where both templates copied from same location 2. **Add template selection to shell scripts** - run.sh and setup.sh now prompt users to choose template - Interactive menu: 1) code (development) or 2) create (content) - Passes template to genie init command 3. **Expand executor list from 3 to 9** - Added: gemini, cursor, qwen_code, amp, copilot - Dynamic generation from EXECUTOR_LABELS mapping - Matches Forge profile names in init.ts Fixes issues identified in init exploration: - Template parameter was ignored (paths.ts:29-32) - Shell scripts didn't prompt for template - Executor registry was hardcoded instead of comprehensive Related: #148, #150, #151 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

workflow_dispatch was checking out main instead of the tag, causing version mismatch when package.json is on dev branch. Now explicitly checks out the tag after fetching it. Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Removed --label flag that doesn't exist (causes errors) Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

…olation) (automagik-forge 1f61a2d9) **Critical Pattern Violation Detected:** Base Genie (main conversation) created Forge task + started attempt for bug #168 (graceful shutdown), then immediately started implementing the fix itself in main workspace. **This Must Never Happen:** - Once Forge task attempt starts → Genie STOPS - Work happens in isolated worktree (Forge executor) - Genie = orchestrator, NOT implementor **Learn Task Goals:** 1. Document this violation pattern clearly 2. Create enforcement checklist for Genie 3. Add to AGENTS.md or skills/ as new rule 4. Ensure this pattern is prevented architecturally **Context:** - Violation: Base Genie editing .genie/cli/src/genie-cli.ts after starting attempt b51db539 - Should have: Waited for Forge executor to complete work - Root cause: Unclear boundaries between orchestration vs execution **Amendment Candidate:** This might be Amendment #4 or #5 in the Seven Amendments framework. **Deliverables:** 1. New skill file: `.genie/skills/orchestration-boundary-protocol.md` 2. Update AGENTS.md with Amendment #4 or #5 3. Create enforcement checklist for Genie base conversation 4. Document architectural patterns to prevent this violation **Evidence Storage:** - Pattern violation: Bug #168, task attempt b51db539 - Session: 2025-10-21 ~05:45 UTC - Files affected: .genie/cli/src/genie-cli.ts (should NOT have been edited by Genie) Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Templates were missing from published package, causing 'Template missing' error during genie init. Added .genie/code/**/* and .genie/create/**/* to files array. Fixes user report from raphaelcrosa on macOS. Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Three critical fixes: 1. **Templates missing from npm package** - Added .genie/code/**/* and .genie/create/**/* to files array - Fixes 'Template missing' error reported by raphaelcrosa - Templates now correctly packaged and distributed 2. **Smart run.sh launcher** - Auto-checks for updates on every run - Offers global install upfront with benefits explanation - Enhanced template selection with descriptions and emojis - Streamlined from 3 steps to one-command setup - Reduces friction: ./run.sh does everything 3. **Improved setup.sh wizard** - Consistent template selection menu - Better UX with colored output and progress indicators - Clear next steps after installation User experience improvements: - One command: ./run.sh (handles everything) - Update notifications built-in - Template selection with full descriptions - Visual improvements with emojis and formatting Fixes user report from raphaelcrosa@MacBook Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

### Fixed - Init now copies root agents + chosen collective (preserving structure) - Both root and code/create collectives discoverable - Added git initialization check before Forge startup - code/install and other collective agents now found correctly ### Changes - copyTemplateGenie → copyTemplateFiles with two-stage copy: 1. Copy root agents/workflows from package .genie/ 2. Copy chosen collective DIRECTORY (not contents) - Added git.git check, prompts to initialize if missing - Added promptYesNo helper for user prompts ### Result Structure User's .genie/ now has: - agents/ (root: analyze, forge, learn, review, wish) - code/ or create/ (collective with agents, skills, workflows) - Both collectives discoverable by agent-resolver fixes #172 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

**Problem:** Template menu wasn't showing during `run.sh`/`setup.sh` execution. Command substitution `template=$(select_template)` was capturing ALL stdout including menu display text. **Root Cause:** When using `$(command)`, bash captures ALL stdout. The menu echo statements were being captured into the variable instead of displaying to the user. **Solution:** Redirect menu display to stderr (`>&2`) so only the final choice goes to stdout and gets captured. **Changes:** - run.sh: Added `>&2` to all menu echo statements in select_template() - setup.sh: Added `>&2` to all menu echo statements in select_template() **Result:** Menu displays on screen, only "code" or "create" gets captured. Fixes #174 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

**Progress:** 1. ✅ Fixed shell script template selection menu (PR #175) 2. ✅ Created Ink wizard component (init-wizard.tsx) 3. ✅ Created Ink chat component for MCP conversation (install-chat.tsx) 4. ✅ Updated run.sh to auto-start genie server after init **Next Steps:** - Rewrite init.ts to integrate Ink components - Remove wrong "Genie Sessions" Forge project creation - Test complete onboarding flow **Architecture:** run.sh → genie init (Ink wizard) → Install chat (MCP) → genie server starts → Browser opens Relates to #174 (template menu bug) Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

## Summary Redesign Genie init flow to use Ink for interactive onboarding with MCP install agent chat. Scaffolded components but blocked on ESM/CommonJS module conflicts. Core architecture ready. ## Changes ### Fixed Wrong Behavior - **Removed runInstallViaCli() calls** (.genie/cli/src/commands/init.ts:136,263) - Was spawning separate process that incorrectly created "Genie Sessions" Forge project - Install agent should work WITHOUT creating wrong Forge project - Added TODO comments for future Ink chat integration ### Architecture: Interactive Onboarding Flow **Target Flow:** 1. run.sh → pnpm install -g automagik-genie@next 2. genie init → Ink wizard (template/executor/model selection) 3. Ink chat → MCP conversation with install agent 4. Install completes → genie server auto-starts 5. Browser opens automatically (already working) ### Scaffolded Components (WIP - ESM Issues) - **init-wizard.tsx.wip** - Fast interactive wizard - Template selection (code/create) - Executor selection (codex/claude/etc) - Model input (optional) - Git initialization check - ESC to cancel - **install-chat.tsx.wip** - Real-time MCP chat - Streaming conversation with install agent - Message history display - Ctrl+N: Skip/continue - Ctrl+Shift+N: Restart install - ESC: Exit ### Updated run.sh - Auto-start genie server after init completes - Better error handling and user feedback - Proper exit codes ### TypeScript Fixes - Applied `as TemplateType` casts (init.ts:95,199,201) - Commented out Ink imports pending ESM resolution - Stubbed interactive mode with helpful message - Build passes successfully ## Blocked: ESM/CommonJS Module Conflict **Error:** ``` TS1479: The current file is a CommonJS module whose imports will produce 'require' calls; however, the referenced file is an ECMAScript module and cannot be imported with 'require' ``` **Ink packages are ESM, Genie CLI is CommonJS** **Temporary Solution:** - Renamed .tsx → .tsx.wip to unblock builds - Preserved complete implementation for future integration **Resolution Options:** 1. Convert CLI to ESM (package.json type:"module" + tsconfig updates) 2. Use dynamic imports for Ink components 3. Create separate ESM entry point for interactive mode ## Next Steps 1. Resolve ESM/CommonJS conflicts 2. Enable Ink wizard in init.ts 3. Implement install agent MCP conversation 4. Test complete flow: wizard → chat → server → browser 5. Optimize duplicate install.md files ## Testing - ✅ TypeScript build passes - ✅ run.sh shell script fixes merged (PR #175) - ⏸️ Interactive mode temporarily stubbed - ⏸️ Ink components ready but not integrated **Related:** Shell script template selection bug fixed (PR #175) Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

## Changes ### Integrated Ink-Powered Interactive Wizard - Created `.mjs` ES module version of init-wizard component - Updated `init.ts` to dynamically import .mjs wizard - Excluded `.tsx` files from TypeScript compilation - Used `@ts-expect-error` for runtime .mjs import ### Interactive Flow 1. Git initialization check (if no .git) 2. Template selection (code/create) with descriptions 3. Executor selection (codex/claude/etc) 4. Model input (with smart defaults) 5. Visual feedback with Ink Gradient + Spinner ### Technical Approach - **Problem:** Ink packages are ESM-only, CLI is CommonJS - **Solution:** - Exclude .tsx from build (tsconfig) - Create hand-crafted .mjs ES module - Dynamic import at runtime - No build complexity, works immediately ### User Experience - Beautiful terminal UI with Ink components - Keyboard navigation (↑/↓ arrows, Enter) - ESC to cancel at any point - Smart defaults based on executor - Git detection and prompt ### Files Changed - `.genie/cli/src/commands/init.ts` - Integrated wizard - `.genie/cli/dist/views/init-wizard.mjs` - ES module UI component - `.genie/cli/tsconfig.json` - Exclude .tsx files - `.genie/cli/src/views/init-wizard.tsx` - Source (not compiled) ## Testing Needed - Fresh `genie init` in new directory (no .genie) - Verify all prompts display correctly - Verify selections passed through correctly - Verify git initialization works ## Next - Test in fresh directory - Add install agent chat flow (install-chat.mjs) - Publish new RC Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

- Track Forge child process reference for direct signal handling - Send SIGTERM to Forge process group on shutdown (prevents orphans) - Check for running tasks before shutdown - Prompt user with task URLs if work in progress - Allow cancellation if tasks are running - Fall back to force shutdown on second Ctrl+C Fixes #168 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

- NEVER manually publish RCs after PR merges to main - GitHub Actions automatically bumps version and publishes - Evidence: gh run list shows 'Publish to NPM' workflow - First violation: 2025-10-21 (tried rc.28 after PR #175, rc.29 already published) Updates brain (AGENTS.md) with permanent learning. Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

**Problem:** Token statistics were coded but not displaying in `genie` dashboard because: - REST API `getExecutionProcess()` doesn't include logs - Logs only accessible via WebSocket endpoints - Token parser expected different event format **Solution:** 1. Added WebSocket-based log fetching using Forge's `/raw-logs/ws` endpoint 2. Extract STDOUT content from JsonPatch operations 3. Parse newline-delimited JSON events from STDOUT 4. Added parser for `stream_event` with `message_start` containing usage data 5. Added @types/ws dev dependency for TypeScript support **Changes:** - `.genie/cli/src/lib/token-tracker.ts`: - Replaced REST API call with WebSocket fetcher - Added STDOUT extraction from JsonPatch - Added stream_event parser for Claude Code executor format - Supports multiple executor formats (Claude Code, Codex, etc.) - `package.json`: Added @types/ws@8.18.1 **Tested:** ✅ Successfully extracts token metrics from completed execution processes ✅ Metrics: 4,072 input + 2,804 output + 450,125 cached = 6,876 total tokens Token statistics now display in: - Live dashboard (compact format during genie command) - Shutdown report (detailed format on exit) fixes #176 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: Token usage statistics now display via WebSocket log fetching

PRs #1626/1627/1628 enforce UUID-only agents.id post migration 061. Tests in resolveSpawnIdentity (agents.test.ts) and tui-spawn-dx integration (Group 8) assert the legacy id=name contract that those PRs deliberately broke. Skipping with TODO pointer to retire-session-names wish #175 which is rewriting them holistically (G1 + G2 in flight, see tasks #176/#177). This unblocks dev CI so the auto-version bump can publish v4.260503.11+ with the spawn pipeline UUID fixes that the production server needs. Validation: bun run typecheck clean; bun run lint 0 errors. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

namastex888 and others added 30 commits October 21, 2025 16:37

fix: remove pnpm version conflict in GitHub Action

dd541b8

Use packageManager field from package.json instead of specifying version in workflow. fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: merge main into dev, resolve pnpm version conflict

3641a21

Keep dev's fix (no explicit version, use packageManager from package.json) fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: add contents write permission to workflow for git push

586fc10

GitHub Actions needs explicit permission to push commits back to the repository. fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: merge main into dev, keep pull-requests write permission

7384746

fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: merge main into dev, keep idempotent workflow

640603e

fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: remove release label from auto-PR (label doesn't exist)

24427bf

The workflow successfully creates GitHub releases but was failing when trying to add a 'release' label that doesn't exist in the repository. fixes #161 Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: pre-release v2.4.2-rc.22

003bb0a

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: checkout tag in publish workflow

f1ac4d7

workflow_dispatch was checking out main instead of the tag, causing version mismatch when package.json is on dev branch. Now explicitly checks out the tag after fetching it. Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: resolve merge conflict in publish-rc-on-merge workflow

6986de3

Removed --label flag that doesn't exist (causes errors) Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: bump automagik-forge to 0.4.2

bf09d3e

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

fix: resolve merge conflicts - keep smart launcher versions

7bcc62e

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Merge branch 'main' into dev

a6061aa

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: bump version to 2.4.2-rc.26

3d27856

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: bump version to 2.4.2-rc.27

2811715

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

chore: pre-release v2.4.2-rc.28

a09217b

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

Merge branch 'main' into dev - resolved conflicts (kept Ink wizard)

d4836ca

Co-authored-by: Automagik Genie 🧞 <genie@namastex.ai>

namastex888 merged commit 8754233 into main Oct 21, 2025
1 of 3 checks passed

namastex888 added a commit that referenced this pull request Nov 3, 2025

Merge pull request #177 from namastexlabs/dev

ed15b58

fix: Token usage statistics now display via WebSocket log fetching

namastex888 mentioned this pull request May 4, 2026

test(spawn): skip legacy id=name tests pending retire-session-names #175 #1630

Merged

4 tasks

This was referenced May 7, 2026

agent-registry register() ON CONFLICT silently drops native_team_enabled + provider — every claude worker misrouted to tmux send-keys #1682

Closed

fix(registry): preserve native_team_enabled + provider across ON CONFLICT in register() #1684

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Token usage statistics now display via WebSocket log fetching#177

fix: Token usage statistics now display via WebSocket log fetching#177
namastex888 merged 31 commits into
mainfrom
dev

namastex888 commented Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

namastex888 commented Oct 21, 2025

Summary

Problem

Solution

Changes

Testing

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant