[Feature]: Desktop — capability handlers for macOS UI control (screenshot, click, keys, AX, AppleScript)

[theonlyhennygod]

Summary

Implement the actual macOS operations the agent can dispatch over /ws/nodes once the persistent NodeClient is in place. Each handler reads its required permission via the existing primitives in apps/tauri/src/macos/permissions.rs, executes the operation, and returns a structured result.

Problem

The persistent WebSocket and capability advertisement are useless without handlers that actually do the work. We need first-class implementations for the high-value verbs.

Proposal

• New module tree apps/tauri/src/capabilities/:
  • screenshot.rs — uses CGDisplayCreateImage / ScreenCaptureKit; requires Screen Recording.
  • click.rs — synthetic mouse click via CGEventCreateMouseEvent; requires Accessibility.
  • type_keys.rs — synthetic keyboard via CGEventCreateKeyboardEvent; requires Accessibility.
  • read_ax.rs — read UI tree via AXUIElement; requires Accessibility.
  • applescript.rs — osascript -e <script>; requires Automation per-app.
  • notify.rs — UNUserNotificationCenter.add; requires Notifications.
• Each handler:
  1. Calls macos::permissions::check_* to verify the prerequisite.
  2. If denied, returns a structured permission_denied(name) error so the agent can surface "I lost — re-grant?".
  3. On success returns serializable output (base64 PNG for screenshot, {} for clicks, JSON tree for AX read).

Approval gating (per #6321)

• Read-only ops (screenshot, read_ax) auto-approve.
• Risky ops (click, type_keys, applescript) trigger a system dialog the first time per-app and remember per a small allowlist persisted via tauri-plugin-store.

Files

• apps/tauri/src/capabilities/{mod,screenshot,click,type_keys,read_ax,applescript,notify}.rs (new)
• apps/tauri/src/lib.rs — register handlers in the NodeClient dispatch table
• apps/tauri/Cargo.toml — add core-graphics and core-foundation if needed

Acceptance

• From a wscat session: dispatching {type:"invoke", capability:"screenshot"} returns a base64 PNG of the primary display.
• Dispatching {capability:"click", args:{x:100, y:200}} clicks at that coordinate.
• Revoking screen recording in System Settings → next screenshot dispatch returns permission_denied("screen_recording").
• Risky ops trigger the user-approval dialog the first time and remember the choice.

Related

• Depends on Issue A (Device↔Node bridge) and Issue B (NodeClient).
• Approval gating shares infrastructure with [Feature]: Desktop menu-bar chat — supervised tool-approval prompts (parity with #6207) #6321 (supervised tool-approval prompts).

[m13v]

CGDisplayCreateImage is deprecated in Sonoma 14.0+ and returns nil on some Apple Silicon configs, so screenshot needs ScreenCaptureKit from day one, not migrated later. the per-app allowlist also conflates two layers: TCC has per-app entries for osascript Automation, but Accessibility is binary at the framework level, the whole binary is approved or nothing. 'remember per-app' for click/type/read_ax has to live in tauri-plugin-store, none of it maps to OS state. read_ax also needs a walk timeout, AXUIElementCopyAttributeValue blocks on hung apps and Electron trees with thousands of nodes lock the dispatcher.