When to Apply

• User wants to access or interact with external apps (Gmail, Slack, GitHub, Notion, etc.)
• User wants to automate a task using an external service (send email, create issue, post message)
• Building an AI agent or app that integrates with external tools
• Multi-user apps that need per-user connections to external services

Setup

Check if the CLI is installed; if not, install it:

curl -fsSL https://composio.dev/install | bash

After installation, restart your terminal or source your shell config, then authenticate:

composio login       # OAuth; interactive org/project picker (use -y to skip)
composio whoami      # verify org_id, project_id, user_id

For agents without direct browser access: composio login --no-wait | jq to get URL/key, share URL with user, then composio login --key <cli_key> --no-wait once they complete login.

1. Use Apps via Composio CLI

Use this when: The user wants to take action on an external app directly — no code writing needed. The agent uses the CLI to search, connect, and execute tools on behalf of the user.

Key commands (new top-level aliases):

• composio search "<query>" — find tools by use case
• composio execute "<TOOL_SLUG>" -d '{...<input params>}' — execute a tool
• composio link [toolkit] — connect a user account to an app (agents: always use --no-wait for non-interactive mode)
• composio listen — listen for real-time trigger events

Typical workflow: search → link (if needed) → execute

Full reference: Composio CLI Guide

2. Building Apps and Agents with Composio

Use this when: Writing code — an AI agent, app, or backend service that integrates with external tools via the Composio SDK.

Run this first inside the project directory to set up the API key:

composio init

Full reference: Building with Composio

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi API Documentation

API Modules

All auth routes resolve against the local SQLite database. There is no cloud dependency — openloomi is fully self-contained. The remote-auth prefix is historical (the routes once proxied to a cloud server); today they are the canonical local endpoints, and the Claude/Codex plugin bridge uses /api/remote-auth/user as a port-discovery + auth-handshake probe.

This reference covers 131 route handlers under 36 top-level /api/* modules (auto-surveyed from apps/web/app/api/).

Functional Modules

Platform Callback Modules

Each integration platform has its own /api/<platform>/* module:

Endpoints Reference

Auth Module

Messages Module

Files Module

Storage Module

Integrations Module

Platform Callbacks

RAG Module

Workspace Module

AI Module

Chronicle Module

Insight Tabs Module

Chat Insights Module

Memory Module

Native Module

Pet Module

Loop Module (highlights)

24 routes total. Top-level surfaces:

Other Modules (single-route or paired)

Error Handling

Error Response Format

// API errors return standard HTTP status codes
{
  error: string;      // Error message
  code?: string;       // Error code for programmatic handling
  cause?: string;      // Additional context
}

Common Status Codes

AI/Agent Usage

Local API Access

When running openloomi desktop app, the local API server runs on port 3414 (fallback: 3515):

Authentication Token

The auth token is stored at ~/.openloomi/token (base64 encoded JWT). You must decode it before use:

# Decode base64 to get JWT token
TOKEN=$(cat ~/.openloomi/token | base64 -d)

# Verify token contents (decodes JWT payload)
echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | python3 -m json.tool

curl Examples

Important: All authenticated requests require the token to be base64 decoded first.

# Helper: Get decoded token
TOKEN=$(cat ~/.openloomi/token | base64 -d)

# 1. Check AI API status (no auth required)
curl http://localhost:3414/api/ai/chat

# 2. Get current user info (also used by Claude/Codex plugin as a port-discovery probe)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl http://localhost:3414/api/remote-auth/user \
  -H "Authorization: Bearer $TOKEN"

# 3. Create an anonymous guest session (no credentials)
curl -X POST http://localhost:3414/api/remote-auth/guest \
  -H "Content-Type: application/json" \
  -d '{}'

# 4. Chat with AI (streaming)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/ai/chat \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"stream":true}'

# 5. Get chat insights (requires chatId)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl "http://localhost:3414/api/chat-insights?chatId=xxx" \
  -H "Authorization: Bearer $TOKEN"

# 6. Search RAG documents
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/rag/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"search term","limit":5}'

# 7. List workspace skills
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl http://localhost:3414/api/workspace/skills \
  -H "Authorization: Bearer $TOKEN"

# 8. Submit feedback
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/remote-feedback \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"content":"Feedback message","email":"user@example.com"}'

Summary

• 131 route handlers across 22 functional modules + 12 platform callback modules + 2 cross-cutting modules (proxy, db)
• Fully self-contained: all auth, data, AI, and sync run locally — no cloud dependency
• Dual authentication: Session cookies (web) and Bearer tokens (Tauri)
• RESTful JSON APIs with Zod validation
• SWR utilities for client-side data fetching
• OAuth support for Slack, Discord, X, HubSpot, LinkedIn, Notion
• RAG for document retrieval and search
• AI endpoints for chat, images, audio
• Loop for attention loop, decisions, channels, classifier rules
• Pet state mirror (read/write /api/pet/state)

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi Connectors Skill

OpenLoomi Connectors provides access to 7 messaging and productivity platform integrations. It allows AI agents to manage OAuth connections, list connected accounts, check connection status, and disconnect platforms on behalf of the user.

Pairing with the composio skill: This skill covers openloomi's native 7 integrations listed below. For accounts connected through Composio (a broader 1000+ apps surface — e.g. X, LinkedIn, Notion, HubSpot, Linear, Jira, etc.), invoke the composio skill in parallel: use the composio-cli to list connections, or call mcp__composio__COMPOSIO_MANAGE_CONNECTIONS with action: "list". When the user asks "what am I connected to?" or "list my accounts", run both — list-accounts here and the composio connection listing — and present the union. Keep auth, OAuth, and disconnect flows native to each skill.

What is openloomi?

Most AI assistants function as workflow tools—users give commands, they execute tasks, with no persistent knowledge of who you are or what matters to you.

openloomi takes a fundamentally different approach: it operates as a proactive digital partner that watches, learns, remembers, and acts on your behalf. The difference is architectural.

How It Works

When users connect messaging platforms and integrations to openloomi, they sync with permission:

• Raw messages and communications
• Meetings and calendar events
• Emails and tweets
• Voice calls
• Notes and captured ideas

This aggregated data becomes "the single source of truth for openloomi's brain."

The Continuous Sync Loop

openloomi runs a background agent on a continuous sync loop, actively gathering information from all connected sources. An agent without this loop can only respond based on stale context. With it, every conversation—and every moment—makes openloomi smarter and more aligned with you.

Supported Platforms (7)

The CLI list-platforms returns these 7 platforms. Other connectable platforms (Slack, Discord, X, Gmail, Outlook, LinkedIn, Google Calendar, Google Drive, Google Docs, HubSpot, Notion, etc.) are managed via the desktop UI or the composio skill — see "Platform Connection Methods" below for details.

Authentication

The CLI auto-reads your token from ~/.openloomi/token (base64 encoded JWT).

Local API Access

The local API server runs on port 3414 (fallback: 3515). If 3414 is unavailable, try 3515.

API Endpoints

Integration Accounts

GET /api/integrations/accounts - List Connected Accounts

Returns all connected platform accounts for the authenticated user.

curl http://localhost:3414/api/integrations/accounts \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "accounts": [
    {
      "id": "int_xxx",
      "platform": "gmail",
      "externalId": "user@gmail.com",
      "displayName": "My Gmail",
      "status": "active",
      "metadata": {},
      "createdAt": "2024-01-01T00:00:00Z",
      "botId": "bot_xxx"
    }
  ]
}

Note: Each account includes a botId field which is used for send-reply and other bot operations.

OAuth Start Endpoints

GET /api/integrations/slack/oauth/start?userId=<userId> - Start Slack OAuth

Returns the Slack OAuth authorization URL. The CLI opens this URL in the browser for the user to complete authorization.

curl "http://localhost:3414/api/integrations/slack/oauth/start?userId=<userId>"

Response:

{
  "authorizationUrl": "https://slack.com/oauth/v2/authorize?...",
  "state": "userId:uuid"
}

GET /api/integrations/discord/oauth/start?userId=<userId> - Start Discord OAuth

Returns the Discord OAuth authorization URL.

GET /api/integrations/x/oauth/start?userId=<userId> - Start X OAuth

Returns the X/Twitter OAuth authorization URL.

OAuth Exchange Endpoints

GET /api/integrations/slack/oauth/exchange?code=<code>&state=<state> - Exchange Slack Code

Exchange OAuth code for Slack access.

GET /api/integrations/discord/oauth/exchange?code=<code>&state=<state> - Exchange Discord Code

Exchange OAuth code for Discord access.

OAuth Callbacks

DELETE /api/integrations/:id - Disconnect Account

Delete a connected integration account.

curl -X DELETE http://localhost:3414/api/integrations/int_xxx \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "success": true,
  "deletedAccountId": "int_xxx",
  "deletedBotIds": ["bot_xxx"]
}

GET /api/contacts - Query Contacts

Query user contacts with optional filtering and pagination.

curl "http://localhost:3414/api/contacts?name=John&page=1&pageSize=10" \
  -H "Authorization: Bearer $TOKEN"

Parameters:

• name (string, optional) - Filter contacts by name (partial match)
• page (number, default 1) - Page number
• pageSize (number, default 10) - Items per page (max 100)

Response:

{
  "success": true,
  "contacts": [
    {
      "id": "contact_xxx",
      "name": "John Doe",
      "type": "email",
      "botId": "bot_xxx",
      "platform": "gmail"
    }
  ],
  "pagination": {
    "page": 1,
    "pageSize": 10,
    "totalCount": 50,
    "totalPages": 5,
    "hasMore": true,
    "hasPrevious": false
  }
}

POST /api/messages - Send Message

Send a message via a connected platform bot.

curl -X POST http://localhost:3414/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "botId": "bot_xxx",
    "recipients": ["John"],
    "message": "Hello!",
    "subject": "Optional subject"
  }'

Parameters:

• botId (string, required) - The bot ID to send from
• recipients (array, required) - List of recipient names
• message (string, required) - Message content
• subject (string, optional) - Email subject line
• cc (array, optional) - CC recipients
• bcc (array, optional) - BCC recipients

Note: botId is returned by list-accounts in the botId field (different from account id).

Platform Aliases Reference

Aliases are case-insensitive and support both English and Chinese:

Desktop UI

Users can also authorize accounts directly through the openloomi desktop application without using CLI commands.

Adding Account Authorization via Desktop UI

1. Open openloomi desktop app on your computer

2. Navigate to Settings (gear icon in the sidebar or top-right menu)

3. Go to Integrations tab/section

4. Click on the platform you want to connect (e.g., Telegram, Slack, Discord, Gmail, etc.)

5. Follow the platform-specific authorization flow:

   • OAuth platforms (Slack, Discord, X/Twitter): Click "Connect" → you'll be redirected to the platform's authorization page in your browser → Approve the permissions → you'll be redirected back
   • App Password platforms (Gmail, Outlook): Enter your email and app password
   • App Credentials platforms (DingTalk, Feishu, QQ): Enter your appId and appSecret
   • QR/Interactive platforms (WhatsApp, Telegram, iMessage): Scan the QR code or follow the in-app instructions

6. Verify connection — once authorized, the platform will show as "Connected" with a green checkmark

Managing Connected Accounts

• List connected accounts: Settings → Integrations → shows all connected platforms with status
• Disconnect account: Settings → Integrations → click on connected platform → "Disconnect" or remove
• Check status: Connected platforms show green "Active" badge; expired/disconnected shows red "Inactive" badge

CLI Script

Quick Start

# List all supported platforms
node $SKILL_DIR/scripts/openloomi-connectors.cjs list-platforms

# List all connected accounts (includes botId for send-reply)
node $SKILL_DIR/scripts/openloomi-connectors.cjs list-accounts

# Cross-source audit: openloomi-native + composio-linked accounts (run together, present union)
node $SKILL_DIR/scripts/openloomi-connectors.cjs list-accounts
# In parallel, invoke the `composio` skill (e.g. `composio list-connections` via composio-cli,
# or `mcp__composio__COMPOSIO_MANAGE_CONNECTIONS` with action: "list")

# Check connection status for a platform
node $SKILL_DIR/scripts/openloomi-connectors.cjs status telegram

# Connect a platform (opens browser for OAuth)
node $SKILL_DIR/scripts/openloomi-connectors.cjs connect slack

# Disconnect an account by ID
node $SKILL_DIR/scripts/openloomi-connectors.cjs disconnect int_xxx

# Query contacts
node $SKILL_DIR/scripts/openloomi-connectors.cjs query-contacts --name=John --page=1 --pageSize=10

# Send a message (requires botId from list-accounts)
node $SKILL_DIR/scripts/openloomi-connectors.cjs send-reply --botId=bot_xxx --recipients=John --message="Hello!"

Commands

Platform Connection Methods

AI Agent Workflow

Triggered when the user asks about:

1. Connecting a platform - "connect telegram", "link my slack"
2. Listing integrations - "show my connected accounts", "what platforms am I connected to"
3. Checking status - "is my github connected?", "telegram status"
4. Disconnecting - "disconnect my discord", "remove whatsapp"
5. Querying contacts - "show my contacts", "find John in contacts"
6. Sending messages - "send email to John", "reply to that message"
7. Cross-source account audit - "show everything I'm connected to (openloomi + composio)", "list all linked accounts across both" → run list-accounts here and the composio skill in parallel, then present the union

Execution Flow:

1. Identify intent - connect / list / status / disconnect / query-contacts / send-reply
2. Resolve platform - use alias normalization (e.g., gh -> github)
3. Execute command - use Bash tool
4. Format output - report results naturally in user's language

Note on send-reply: The botId is returned by list-accounts in the botId field.

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi Product Features

Use this skill when users ask about openloomi features, usage, or capabilities. Provide accurate and easy-to-understand feature introductions and operation guides.

What is OpenLoomi

openloomi is a Proactive AI Workspace that understands your intent, orchestrates execution, and gets things done. It's not just another AI assistant—it's an innovative AI product that senses business signals, orchestrates tasks autonomously, and tracks and validates results end-to-end.

Core Value Proposition

openloomi transforms how individuals and SMB teams work by:

• Proactive Awareness — Monitors signals across platforms and alerts you before you ask
• Long-Term Memory — Remembers context across months, never forgets commitments
• Autonomous Execution — Not just telling you what to do, but doing it
• Builtin Skills — Rich execution capabilities for every work scenario

Core Capabilities

🧠 Long-Term Memory

Clear recollection, never forgotten. openloomi builds persistent knowledge graphs that remember all important people, events, decisions, and context across sessions and time. Six months later, it still knows your commitments and preferences.

🎯 Noise Filtering

Tells you what you should act on. With hundreds of daily messages, openloomi replaces "information overload" with "priority signals." Filters 95% of noise, focusing your attention on the 5% that truly matters.

⚡ Powerful Engine

Intent understanding, automatic orchestration. When you say "Help me prepare an investor pitch," openloomi automatically understands intent, breaks it into multiple sub-tasks, invokes appropriate Skills, and chains execution.

🔐 Security & Privacy

Your data, your sovereignty. Local-first architecture—your raw data never leaves your device. End-to-end AES-256 encryption, zero-data-training commitment, SOC 2 compliance audit.

🛠️ Builtin Skills

Builtin Skills covering every work scenario, continuously expanding:

• 📊 Data Analysis
• 💻 Code Generation
• 📄 Document Creation
• 🌐 Web Automation
• 🎨 Image Generation
• 📧 Email Writing
• 🔍 Deep Research
• 📊 PPT Creation
• more skills...

Use Cases

🌍 Global Managers

Never miss a critical signal across time zones. Business runs 24/7 globally. openloomi filters time zone and language noise, capturing high-value opportunities while you sleep—wake up to a refined action list.

🧑‍💻 Engineers & Product Teams

Team memory that never decays. Transform discussions scattered across Slack, Jira, and documents into structured knowledge. Auto-generate weekly reports, sync missed context, eliminate "context rot."

🚀 Founders & Sales

One person does the work of many, at scale. openloomi learns your communication style, automatically maintains hundreds of client relationships, follows up on leads, generates personalized proposals—never burns out.

Quick Start

1. Sign Up / Sign In

• Sign up with email and password
• Or sign in directly with your Google or GitHub account

2. Onboarding

First-time users will go through an onboarding flow:

1. Select your role and focus areas
2. Tell openloomi what you'd like it to help with
3. Connect platforms to unlock deeper insights
4. Name your AI assistant

3. Connect Communication Platforms

Click [Connect platform] to complete authorization.

Supported platforms:

• Messaging: Slack, Telegram, Discord, WhatsApp, Weixin, iMessage, QQ, Feishu, DingTalk
• Email: Gmail, Outlook
• Social Media: X (Twitter) — for marketing and content automation
• Other: RSS
• Coming Soon: Google Drive, Microsoft Teams, Notion, HubSpot, Google Calendar

Platform Connection Steps

WhatsApp

1. Click [Connect WhatsApp]
2. Complete authorization via QR code scan or phone pair code
3. Once authorized, openloomi will automatically read your WhatsApp messages to generate long-term events. You can send messages in "Starred Messages" and AI will:

• Read and understand your message content
• Generate smart insights in openloomi
• You can converse with AI about these messages in openloomi

💡 How to use: Open "Starred Messages" in WhatsApp, send a message to yourself, and AI will automatically read and understand it.

Weixin

1. Click [Connect Weixin]
2. Complete authorization via QR code scan

Telegram

1. Click [Connect Telegram] to enter the authorization page in the source settings.
2. Choose a login method:
   • Phone verification: Enter phone number → receive verification code → enter 2FA password if enabled
   • QR code: Scan QR code with Telegram → enter 2FA password if enabled
   • Quick login: If you have the official Telegram desktop app installed locally, you can use your existing session to log in without phone number or verification code
3. Once authorized, openloomi will automatically read your Telegram messages to generate long-term events on the Today page. You can send messages in "Saved Messages" and AI will:

• Read and understand your message content
• Generate smart insights in openloomi
• You can converse with AI about these messages in openloomi

💡 How to use: Open "Saved Messages" in Telegram, send a message to your saved messages, and AI will automatically read and understand it.

Slack

1. Click [Connect Slack] in the integration settings
2. Click [Install openloomi] on the authorization page to add to your workspace
3. Note: Currently only workspace owners can install

Discord

1. Click [Connect Discord] in the integration settings
2. Select the Discord server to install
3. Grant openloomi bot message permissions
4. Note: Only server admins can install

Gmail

1. Click [Connect Gmail] in the integration settings
2. Enter the email address and app password to authorize

RSS

1. Click [RSS] button to enter the RSS integration page
2. Enter a single RSS link, or upload an OPML file for batch import

Feishu

1. Click [Connect Feishu]
2. Enter your Feishu App ID and App Secret
3. Click connect

How to get credentials:

1. Go to Feishu Open Platform
2. Create an enterprise self-built app
3. Enable bot capability
4. Select "Use long connection to receive events"
5. Subscribe to im.message.receive_v1
6. Get App ID and App Secret from the app settings

Required permissions (scopes):

{
  "scopes": {
    "tenant": [
      "aily:file:read",
      "aily:file:write",
      "application:application.app_message_stats.overview:readonly",
      "application:application:self_manage",
      "application:bot.menu:write",
      "cardkit:card:write",
      "contact:user.employee_id:readonly",
      "corehr:file:download",
      "docs:document.content:read",
      "event:ip_list",
      "im:chat",
      "im:chat.access_event.bot_p2p_chat:read",
      "im:chat.members:bot_access",
      "im:message",
      "im:message.group_at_msg:readonly",
      "im:message.group_msg",
      "im:message.p2p_msg:readonly",
      "im:message:readonly",
      "im:message:send_as_bot",
      "im:resource",
      "sheets:spreadsheet",
      "wiki:wiki:readonly"
    ],
    "user": [
      "aily:file:read",
      "aily:file:write",
      "im:chat.access_event.bot_p2p_chat:read",
      "im:chat:read",
      "im:chat:readonly"
    ]
  }
}

X (Twitter)

Connect X (Twitter) to enable marketing automation features.

1. Click [Connect X] to authorize via OAuth

Outlook

1. Click [Connect Gmail] in the integration settings
2. Enter the email address and app password to authorize

Microsoft Teams

Coming soon!

QQBot

1. Click [Connect QQ]
2. Enter your QQ App ID and App Secret
3. Click connect

How to get credentials:

1. Go to QQ Open Platform
2. Create a bot
3. Get App ID and App Secret from the bot settings

DingTalk

DingTalk integration uses a Stream mode bot — a long-lived WebSocket connection with no public IP or domain required.

Before you start — create a DingTalk app:

1. Go to DingTalk Open Platform and sign in
2. Create an enterprise internal app
3. Add the Bot capability and choose Stream mode (long connection)
4. Copy your Client ID (AppKey) and Client Secret (AppSecret)

Connect in openloomi:

1. Click [Connect DingTalk]
2. Enter your Client ID (AppKey) and Client Secret (AppSecret)
3. Click Connect

💡 Stream mode means openloomi connects directly via WebSocket — no server or domain setup needed.

iMessage

iMessage integration is only available on macOS.

1. Click [Connect iMessage]
2. Grant the required permissions:
   • Full Disk Access - Required to read iMessage database
   • Automation Permission - Required to send messages
3. Enter a display name for your iMessage account
4. Click connect

How to grant permissions:

• Go to System Settings > Privacy & Security > Full Disk Access
• Add the running app (Terminal, Node, or openloomi)
• Restart the app after granting permissions

💡 Your message data stays on your local device. openloomi only reads recent messages when you use it to generate insights.

Desktop App

openloomi also offers a desktop app (macOS, Linux, and Windows) that provides a native local experience.

Download & Install

Windows:

1. Download the latest .exe installer from GitHub Releases
2. Run the installer — if Windows SmartScreen shows a warning, click "More info" then "Run anyway". This is normal for new applications without a long code-signing reputation; openloomi is open-source and the code is publicly verifiable
3. After the first run, SmartScreen typically bypasses automatically on subsequent updates

Winget (Coming Soon):

winget install openloomi.openloomi

💡 You only need to complete the SmartScreen step once per machine.

macOS:

Download the latest .dmg from GitHub Releases and drag openloomi to your Applications folder.

Linux:

Download the latest .AppImage or .deb from GitHub Releases.

Important: App Must Be Running

To use MessageApp conversations and scheduled automation tasks in the desktop app:

• The app must be open and running
• The computer must be turned on and not in sleep mode

If the desktop app is closed or the computer is asleep, conversations and scheduled tasks will not execute.

Local Data Storage

All data in the desktop app — including messages, conversations, scheduled tasks, and settings — is stored locally on your device via SQLite. No app data is sent to or stored on cloud servers.

Permissions

When you first launch openloomi, the system may ask for a few permissions. Each one has a specific purpose — and you can decline any of them. openloomi will continue to work; you'll just lose the feature that requires that permission.

macOS

How to grant or revoke:

1. Open System Settings → Privacy & Security
2. Find the permission category (e.g., Full Disk Access, Automation, Notifications)
3. Toggle openloomi on or off

You can revisit these settings at any time.

Windows

Windows SmartScreen may also show a one-time warning when running the installer. Click "More info" then "Run anyway" — this is normal for open-source software. You only need to do this once per machine.

Linux

No special permissions are required on Linux. openloomi uses the standard desktop notification system (libnotify) to send alerts — if you have granted notification permissions to other apps, openloomi will use them automatically.

Conversation Features

Just type your questions or requests in the chat box, and openloomi will help you find answers.

Example Questions

• "What is openloomi"
• "How to use openloomi"
• "Summarize yesterday's to-dos"
• "Today's important news"
• "What are my contacts"
• "Randomly send 'Hello' to 3 contacts on Gmail"
• "What progress have we made with the XX project this past week?"

Features

• Project collaboration queries - Ask about project progress
• Weekly report generation - Request weekly reports on all project progress
• Web browsing - Have openloomi browse for latest product info
• New conversation creation - Start new conversations around specific topics
• History - View conversation history
• Source References - See exactly which messages/conversations openloomi's answers come from, who was involved, and when they occurred
• Artifacts - openloomi can generate visual artifacts: mind maps, flowcharts, charts, roadmaps, surveys, and documents. Preview and interact with them directly in chat
• File Analysis - Upload files (PDF, images, etc.) in chat and ask openloomi to analyze, summarize, or extract information
• Deep Dive - For certain topics, continue exploring with follow-up questions, detail requests, or scope narrowing
• Topic-Based Chats - Create new conversations around specific topics, review past discussions, build persistent context over time

Chat via Messaging Apps

You can also interact with openloomi directly through your connected messaging apps. Once connected, openloomi becomes your AI assistant within those platforms.

After connecting (e.g., Telegram), just send a message to Saved Messages and openloomi will respond naturally.

Action Features

openloomi can generate action items for various scenarios.

How to View

1. Click the Action button in the event details
2. Or view in the unified Action panel

Features

• To-do display - Show TODOs in the unified panel
• Quick action suggestions - openloomi suggests clickable quick actions
• Detailed information - Fill in sender, recipient, content and attachments
• AI content generation/translation - Auto-generate or translate content
• Message replies - Click reply button in understanding detail view, openloomi will generate a reply

Smart Insights

openloomi automatically analyzes your conversations to extract valuable information.

Automatically Extracted Content

• ✅ To-dos - Tasks to complete, deadlines
• 📈 Project progress - Status updates, milestones
• 🎯 Important decisions - Meeting decisions, key choices
• ⚠️ Risk alerts - Issues to watch out for
• 📅 Timeline - Event development脉络

How to Use

1. View - Click on an event in the left menu
2. Categorize - Mark as: Urgent, Important, Monitor, Archive
3. Add to-do - Add tasks directly in the details
4. Timeline - View event development timeline

Event Management

Events are automatically organized into groups based on their status:

• Opportunities - New events that need attention
• In Progress - Events you're currently working on
• Waiting on Others - Events pending response
• Done - Completed events

You can drag and drop events between groups to manually categorize them. You can also multi-select multiple events for bulk operations (Mark as Done, Archive, Delete).

Event Detail

Click on any event card to open its detail view:

• Event title and description - Full context of the event
• Related messages - All conversation threads involved
• Participants - Everyone included in the event
• Timeline - Chronological activity log
• Notes - Add personal text notes for reference
• Attachments - Upload documents, PDFs, images to keep related materials in one place

Event Conversation

You can chat directly with any event to ask questions and get AI insights within that context.

Sample questions:

• "What is this event about?"
• "Who are the key people involved?"
• "What decisions were made?"

Event Actions

• Source Reply - Reply directly to messages within an event. Use AI suggestions to generate a reply based on conversation context, or use AI Translation to translate to English or Chinese
• AI Polishing - Improve grammar, wording, and tone (formal, casual, friendly)
• Send to Platform - Send your reply directly to the original platform (Telegram, Discord, etc.)

Common Queries

"What's on my to-do list today?"
"How is the XX project progressing?"
"What important messages were there last week?"

Settings

How to access: Click the settings button in the profile panel

Soul

Define your AI assistant's personality and communication style.

Description

Customize how your AI assistant describes itself to others.

Contexts

Configure which data sources and contexts your AI assistant can access. Context types include:

• System - System notifications and status updates
• Event - Grouped communications and projects
• Scheduled Task - Time-based tasks and reminders
• Knowledge - Uploaded documents and reference materials

You can also create custom context tabs with your own name, description, icon, color, and priority. Use keywords to enable automatic categorization.

Interests

Customize what to follow — specific people or topics/projects. For each, you can set:

• Notification Level: All messages, Only @me, or Nothing
• AI Summary: Enable AI-generated summaries of their messages
• Auto-archive: Automatically archive related messages

Connectors

Manage all your connected platforms and services in one place.

Disconnect / Revoke Access

If you no longer want openloomi to access a connected platform, you can disconnect it at any time:

1. Go to Settings → Connectors
2. Find the platform you want to disconnect
3. Click [Disconnect] or the remove (×) button
4. Confirm the action

Once disconnected:

• openloomi will immediately stop reading new messages from that platform
• Previously synced data is retained until manually deleted
• You can reconnect at any time by repeating the connection steps

💡 Tip: Before disconnecting, you may want to review what data has been synced in the Privacy & Security settings.

Language

Change the language used in openloomi.

Search

Search across all your messages, files, and conversations to find exactly what you need.

Scheduled Tasks

Have AI automatically execute tasks at specified times.

How to Create

1. Go to Agent/Automation page
2. Click "New Task"
3. Fill in task information:

Schedule Types

• Cron Expression — Flexible scheduling with cron syntax (e.g., 0 9 * * * = every day at 9am, 0 9 * * 1 = every Monday at 9am)
• Interval — Run every X minutes/hours (e.g., every 30 minutes, every 2 hours)
• One-Time — Run once at a specific date and time

Manage Tasks

• Enable/Disable — Turn tasks on or off
• Run Now — Execute immediately without waiting for the scheduled time
• View History — See past execution results, success/failure status, and output logs
• Edit — Modify task configuration
• Delete — Remove a task

Example Use Cases

• Daily News Summary: "Search latest AI news, summarize top 5 stories and email to me" — every morning at 8am
• Weekly Report: "Generate weekly report on all project progress" — every Friday at 5pm
• Periodic Reminder: "Check calendar for upcoming meetings, remind me 15 minutes before" — every 30 minutes

Knowledge Base

After uploading documents, you can ask AI questions about them directly.

How to Use

1. Upload documents - Upload PDF, Word, text, etc. in settings
2. Ask questions - Ask AI "What's in the document about XXX?"
3. Get full content - View the complete document when needed

Privacy Policy

Your privacy matters. Our Privacy Policy explains in detail how we collect, use, store, and protect your data — including what data we access, how it's encrypted, how long we retain it, and your rights to access, export, or delete it at any time.

Privacy & Security

Your data, your sovereignty. openloomi puts privacy and control first—you never need to trade data sovereignty for intelligence.

🔐 Our Privacy Principles

Local-First Architecture

Your original messages and files stay on your device. openloomi only accesses the minimum data needed to complete tasks—no unnecessary uploads.

• Raw data never leaves your local environment
• Only processed results are transmitted when needed
• Complete control over what data openloomi can access

End-to-End Encryption

All authorized data is encrypted with AES-256 industry standard, processed in hardware-isolated trusted execution environments.

• AES-256 encryption for all stored data
• Hardware-isolated trusted execution environments
• Zero visibility into your raw data

Zero Training Commitment

Your data will never be used to train public AI models—now or ever. In writing, legally guaranteed.

• Explicit legal commitment in our terms of service
• No use of your data for model training
• Full transparency on data usage

Auditable Compliance

All critical operations are logged in tamper-proof audit logs. SOC 2 compliance certification in progress.

• Complete audit trail of all data access
• SOC 2 compliance certification
• Enterprise-grade security standards

🔑 Your Data, Your Control

openloomi gives you complete control over your data:

• Access Controls: Choose exactly which platforms and data openloomi can access
• Data Retention: Configure how long openloomi retains processed data
• Delete Anytime: Delete all your data at any time with one click
• Export: Export all your data in standard formats

We believe privacy and intelligence aren't mutually exclusive. You should never have to choose between powerful AI capabilities and data control. openloomi delivers both.

FAQ

Q: How do I use openloomi?

A: It's simple!

1. Start a conversation in openloomi
2. Describe your needs directly
3. AI will understand and execute

Q: How do I connect Telegram?

A:

1. Click [Connect Telegram]
2. Choose phone verification or QR code login

Q: What are Insights?

A: Automatically extracted important information from your conversations, including to-dos, project progress, decision records, etc.

Q: How do I create automation tasks?

A:

1. Go to Agent/Automation page
2. Click "New Task"
3. Fill in task name and description
4. Set execution time
5. Save

Showcase: openloomi in Action

Real-world examples showing how openloomi works across different use cases.

Chat Scenarios

Office: Smart Meeting Coordination

Scenario: A team needs to schedule a cross-timezone meeting.

User: Hey, find a time this week when everyone is available in the afternoon

openloomi:
📅 Found available slots for 5 team members:

| Time (PT) | Wed | Thu | Fri |
|-----------|-----|-----|-----|
| 2-3pm     | ✅  | ✅  | ❌ |
| 3-4pm     | ✅  | ✅  | ✅ |
| 4-5pm     | ❌  | ✅  | ✅ |

Recommended: Thursday 3-4pm (everyone available)
→ Want me to create a Google Calendar meeting?

What it shows: Google Calendar integration, multi-account availability lookup, natural language scheduling

R&D: Code Review Context

Scenario: A developer gets PR feedback and needs full context.

User: What's the status of PR #247?

openloomi:
📋 PR #247 Review Summary:

🔍 Related Discussion:
- @john: Suggested refactoring auth logic (comment #3)
- @sarah: Requested additional unit tests
- @mike: Approved with "LGTM"

📎 Linked Slack Threads:
#engineering: "Auth module refactoring plan" (3 messages)

📝 Extracted Tasks:
- [ ] Add auth test cases
- [ ] Update API documentation
- Due: This Friday

→ Want me to generate suggested changes?

What it shows: Cross-platform context aggregation, task extraction, timeline tracking

Marketing: Campaign Coordination

Scenario: Marketing team coordinating a product launch.

User: Is our Wednesday product launch ready?

openloomi:
🚀 Product Launch Status:

| Item | Status | Owner |
|------|--------|-------|
| Website update | ✅ Done | @marketing |
| Social posts | 🔄 In Progress | @social |
| Email blast | ⏳ Pending | @email |
| KOL outreach | ❌ Blocked | @partnerships |

⚠️ Blockers:
- KOL partnership awaiting legal approval
- Waiting for final product documentation

→ Want me to remind the team?

What it shows: Task status tracking, cross-team coordination, blocker identification

Collaboration: Project Kickoff

Scenario: Starting a new project with rapid setup.

User: Create a new client project called Acme Corp

openloomi:
✅ Project Acme Corp created!

Auto-completed:
📁 Google Drive folders
  - /Acme Corp/Contracts
  - /Acme Corp/Meeting Notes
  - /Acme Corp/Requirements

💬 Slack channels
  - #acme-general (project overview)
  - #acme-dev (technical)
  - #acme-support (client support)

📅 Shared calendar
  - Kickoff meeting (tomorrow 10am)
  - Weekly standup (Fridays 3pm)

👥 Team invitations sent

What it shows: Multi-platform automation, workflow orchestration

Tracking Scenarios

Proactive: High-Priority Alerts

Scenario: System proactively identifies items needing attention.

📊 Today's Insights - March 12, 2026

🔥 Urgent (3)
├── PR #251 needs review - waiting 4 hours
├── Support tickets piling up - 12 unprocessed
└── Server alert - CPU at 95%

⭐ Important (5)
├── Q1 budget approval - due tomorrow
├── Acme contract renewal - 30 days left
└── ...

📋 Todo Tasks (8)
├── Reply to John's technical question - due today
├── Approve 3 team PTO requests
└── ...

→ View All | Mark Read | Remind Later

What it shows: Importance/urgency ranking, unreplied tracking, deadline reminders

Auto-Managed: Scheduled Refresh

Scenario: System automatically keeps information up to date.

⏰ Scheduled jobs running...

✅ RSS Feeds Updated
├── Tech News (5 new articles)
├── Product Management (2 new articles)
└── Competitor Updates (8 new articles)

✅ Insights Refreshed
├── Refresh interval: 30 minutes (high activity)
├── New insights: 6
└── Archived: 2

✅ Email Automation Triggered
├── User signed up 7 days ago → onboarding email
├── Free quota running low → upgrade reminder
└── Inactive 30 days → re-engagement email

What it shows: Scheduled RSS fetching, automated insight refresh, user lifecycle automation

IM Platform Scenarios

Feishu: Enterprise Workflow

Scenario: Enterprise workflow automation.

Feishu Group:
User: Requesting PTO March 20-22

openloomi:
📝 PTO Request

| Field | Content |
|-------|---------|
| Applicant | @john |
| Dates | March 20-22 (3 days) |
| Type | Annual Leave |
| End Date | N/A |

Approver: @manager

[Approve] [Reject] [Request Info]

What it shows: Approval workflows, message handling, document integration

WhatsApp: Personal Assistant

Scenario: Managing personal tasks via WhatsApp.

WhatsApp conversation:
User: Set a reminder for tomorrow at 2pm

openloomi: ✅ Reminder set

⏰ Tomorrow 2:00pm
📌 Reminder: Call with John
📋 Context: Discussing Q1 proposal
   Client: john@acme.com
   Budget: $50,000

What it shows: QR pairing, status polling, reminder management

Discord: Community Management

Scenario: Automated community operations.

New member joins Discord server

openloomi Welcome Bot:
👋 Welcome @newuser!

I'm openloomi, your community assistant:

📚 Resources
- #announcements - Latest updates
- #help - Get support
- #showcase - Share projects

⚡ Quick Start
- [Introduce Yourself] Share your background
- [Set Up Profile] Customize your info
- [Browse Channels] Explore the community

Ask me anything!

What it shows: New member onboarding, FAQ automation, message handling

🚀 Industry Intelligence Automation

Scenario: Automated AI product news monitoring with daily briefings.

User: Help me set up an industry intelligence automation

openloomi:
✅ Setting up your AI Product Daily Digest Assistant!

Step 1/4: Create Scheduled Task
────────────────────────────────
Task Name: AI Product Daily Digest
Schedule: Daily at 9 AM (Cron: 0 9 * * *)
Description:
1. Scrape X, Reddit, Product Hunt for AI product news
2. Summarize top 10 stories
3. Record to timeline as "AI Product Daily Updates"
4. Send briefing via Telegram

Step 2/4: Connect Data Sources
────────────────────────────────
✅ X (Twitter) - Connected
✅ Reddit - Connected
✅ Product Hunt - Connected
✅ Telegram - Connected

Step 3/4: Task Created
────────────────────────────────
🎉 Your automation is now active!

Next run: Tomorrow at 9:00 AM

Step 4/4: What You'll Receive
────────────────────────────────
☀️ Daily Telegram Briefing:
• Top AI product launches
• Trending discussions
• Engagement metrics

📰 Timeline Event:
• "AI 产品要闻每日更新"
• Full context for follow-up questions

🔮 Coming Soon:
• Visual dashboards
• Team sharing to Slack

What it shows: Scheduled tasks, multi-platform scraping, automated briefings, timeline recording

Example: Daily Briefing Output

☀️ AI Product Daily Digest - March 15, 2026

🔥 Top 5 AI Product Launches Today:

1. 🎨 Claude Art (Product Hunt)
   AI image generation with style transfer
   247 upvotes

2. 💻 Devin 2.0 (X)
   AI coding assistant v2.0
   1.2K retweets

3. 🔧 LangChain v1.0 (Reddit)
   Major agent framework update
   89 upvotes

📈 Trend Summary:
- Image Generation: 🔥 Hot
- AI Coding: 📈 Growing

[View Full] [Create Follow-up] [Share]

What it shows: Multi-source aggregation, smart summarization, actionable outputs

Reference

• openloomi website: https://openloomi.ai
• openloomi documents: https://openloomi.ai/docs

OpenLoomi Hooks Sub-skill

The plugin ships a hook bundle in hooks/hooks.json but never installs it automatically. Users must explicitly run one of:

• /openloomi:hooks install — merge the bundle into ~/.claude/settings.json
• /openloomi:hooks uninstall — strip only the plugin's block (other plugins' hooks are preserved)
• /openloomi:hooks status — report whether the bundle is currently active

Under the hood this calls:

node ${CLAUDE_PLUGIN_ROOT}/scripts/loomi-bridge.mjs install-hooks
node ${CLAUDE_PLUGIN_ROOT}/scripts/loomi-bridge.mjs uninstall-hooks
node ${CLAUDE_PLUGIN_ROOT}/scripts/loomi-bridge.mjs hooks-status

Safety guarantees

• install-hooks is merge-no-overwrite: only adds the plugin's block under hooks.__openloomi_claude_plugin_hooks__. Existing hooks for other plugins are not touched.
• uninstall-hooks removes only the plugin-marked block; rerunning it after the first time reports removed: false, never deletes anything else's hooks.
• The merge is atomic (writeFile to a temp file, then rename).
• The Stop hook always exits 0 — archive failures are reported as JSON {continue: true, _openloomi: {archive: "skipped", reason: ...}} so they can never block Claude Code's response stream.

Suggested user prompt

This plugin can mirror Claude Code's lifecycle onto your Loomi Pet. To enable this, run /openloomi:hooks install. To disable, run /openloomi:hooks uninstall. The plugin never modifies your ~/.claude/settings.json without explicit consent.

OpenLoomi Install Sub-skill

This sub-skill is auto-loaded when the user wants to install or fix OpenLoomi on their machine. It composes loomi-bridge operations and never downloads or executes anything outside the plugin's own scripts.

Quick workflow

1. node ${CLAUDE_PLUGIN_ROOT}/scripts/loomi-bridge.mjs setup-status --json
2. Based on nextAction / reason:

Reminder: secrets contract

• This plugin never handles AI provider API keys. AI provider configuration lives in the OpenLoomi runtime itself; the runtime detects the user's local claude CLI auth on its own.
• If the user pastes a key into chat, redact it: do NOT echo it back, and tell them to remove it from chat history.
• Never pass --api-key as an argv flag — there's no such flag in the plugin, by design.

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi Memory Skill

OpenLoomi Memory is a personal knowledge management tool that searches and manages three types of information:

Overview

openloomi Memory is built on a unique architectural principle: instead of treating memory as an afterthought, it's the foundation.

How it works with Connectors: Before memory can search your data, you need to connect your platforms using the openloomi-connectors skill. Connectors handles OAuth authentication and integration setup for 26 platforms (Telegram, WhatsApp, Slack, Discord, Gmail, Outlook, Twitter/X, WeChat, and more). Once connected, openloomi continuously syncs everything with your permission—raw messages, meetings, emails, tweets, calendar events, voice calls, and any notes or ideas you've captured. This aggregated data becomes the single source of truth that powers openloomi's brain.

The memory is layered:

• Raw information: Original messages, files, transcripts
• Information insights: Extracted entities, decisions, key events
• Contextual memory: Recent conversation state
• Knowledge-base memory: Long-term people/projects/preferences knowledge graph

This enables reasoning across both immediate context and deep historical knowledge simultaneously. When you create custom agent roles to handle tasks, this memory acts as the orchestrator—dramatically improving execution quality.

Inside openloomi, memory is layered into multiple levels:

• Raw information: Original messages, files, transcripts
• Information insights: Extracted entities, decisions, key events
• Contextual memory: Recent conversation state
• Knowledge-base memory: Long-term people/projects/preferences knowledge graph

This enables reasoning across both immediate context and deep historical knowledge simultaneously.

Insights include automatic usage tracking—recording view frequency, sources, and calculating value scores to surface the most relevant information. A periodic maintenance system (daily analytics refresh, weekly compaction) keeps insight retrieval accurate and prevents context decay by archiving or removing stale, low-value content.

Authentication

The CLI auto-reads your token from ~/.openloomi/token (base64 encoded JWT).

Local Memory Filesystem

Overview

Memory files are stored locally at ~/.openloomi/data/memory/ and searched via direct filesystem access. This is a read-only operation that performs case-insensitive text search across .md and .json files.

Directory Structure

~/.openloomi/data/memory/
├── chats/           # Chat conversation exports
├── channels/         # Channel memory exports e.g., weixin, telegram, etc.
├── people/          # Person profiles
├── projects/       # Project notes
├── notes/          # General notes
└── strategy/       # Strategy documents

Write Operations

Memory files are plain markdown or JSON stored locally. You can add or delete files directly.

Adding a memory file:

node $SKILL_DIR/scripts/openloomi-memory.cjs add-memory "Content to remember" --file=filename.md --directory=notes

• --file (optional): Filename. If not provided, auto-generated from first line of content.
• --directory (optional): Subdirectory under ~/.openloomi/data/memory/. Created if doesn't exist.

Deleting a memory file:

node $SKILL_DIR/scripts/openloomi-memory.cjs delete-memory filename.md --directory=notes

How search-memory Works

1. Path: ~/.openloomi/data/memory/ (or subdirectory if specified)
2. Search Type: Case-insensitive full-text search
3. Files: Scans .md and .json files recursively (max depth 5)
4. Matching: Each line is searched; returns first match per file
5. Output: File path, line number, and line preview (first 200 chars)

Example Output

{
  "results": [
    {
      "file": "people/boss.md",
      "line": 42,
      "preview": "My boss John mentioned the deadline is next Friday"
    },
    {
      "file": "projects/app/notes.md",
      "line": 10,
      "preview": "Boss wants the app launched by end of month"
    }
  ],
  "total": 2
}

API Endpoints

Knowledge Base (RAG)

POST /api/rag/search - Search Documents

Semantic search of uploaded documents using embeddings.

curl -X POST http://localhost:3414/api/rag/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "project plan", "limit": 5}'

Parameters:

• query (string, required) - Search query
• limit (number, default 5) - Max results to return

Response:

{
  "results": [
    {
      "id": "doc_xxx",
      "title": "Project Document",
      "content": "...",
      "score": 0.95
    }
  ]
}

GET /api/rag/documents - List Documents

List all documents in the knowledge base.

curl http://localhost:3414/api/rag/documents?limit=50 \
  -H "Authorization: Bearer $TOKEN"

Parameters:

• limit (number, default 50) - Max results to return

Response:

{
  "documents": [
    {
      "id": "doc_xxx",
      "name": "document.pdf",
      "type": "pdf",
      "size": 102400,
      "createdAt": "2024-01-01T00:00:00Z"
    }
  ],
  "total": 10
}

GET /api/rag/documents/[id] - Get Document

Get a single document by ID.

curl http://localhost:3414/api/rag/documents/doc_xxx \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "id": "doc_xxx",
  "name": "Project Document.pdf",
  "type": "pdf",
  "size": 102400,
  "content": "Document text content...",
  "createdAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-01T00:00:00Z"
}

Insights

Insights are structured information extracted from chat history, such as key decisions, action items, and relationship notes. Each insight belongs to one or more groups (channels/platforms) like gmail, telegram, whatsapp, slack, discord, linkedin, twitter, etc.

GET /api/insights - List Insights

List all insights from a time period.

curl "http://localhost:3414/api/insights?days=7&limit=50" \
  -H "Authorization: Bearer $TOKEN"

Parameters:

• days (number, default 7) - Look back period in days
• limit (number, default 50) - Max results to return

Insight Structure: Each insight contains a groups field—an array of channel identifiers indicating which platform(s) the insight came from:

{
  "id": "insight_xxx",
  "chatId": "chat_xxx",
  "type": "decision",
  "content": "John sent an email about the project deadline",
  "groups": ["gmail"],
  "people": ["John"],
  "time": "2024-01-01T00:00:00Z",
  "createdAt": "2024-01-01T00:00:00Z"
}

Insight Types:

Common Channel Groups:

POST /api/insights - Create Insight

Create a new insight manually.

curl -X POST http://localhost:3414/api/insights \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type": "preference", "content": "I prefer Americano coffee", "groups": ["whatsapp"]}'

Parameters:

• type (string, required) - Insight type (decision, action_item, note, preference, relationship, event)
• content (string, required) - The insight text
• groups (array, optional) - Channel groups to associate with
• people (array, optional) - People mentioned in the insight

Response:

{
  "id": "insight_xxx",
  "type": "preference",
  "content": "I prefer Americano coffee",
  "groups": ["whatsapp"],
  "createdAt": "2024-01-01T00:00:00Z"
}

PUT /api/insights/[id] - Update Insight

Partial update an existing insight. Arrays (details, timeline, insights) are appended to, not replaced.

curl -X PUT http://localhost:3414/api/insights/insight_xxx \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "updates": {
      "description": "Updated description",
      "details": [{"content": "User mentioned new preference", "person": "User"}],
      "timeline": [{"summary": "Progress update", "label": "Update"}]
    }
  }'

Update Fields:

• title - New title
• description - New description
• importance - Important, General, Not Important
• urgency - As soon as possible, Within 24 hours, Not urgent, General
• details - Array of detail objects (appended to existing)
• timeline - Array of timeline events (appended to existing)
• myTasks - Array of task objects
• groups - Array of group tags (replaced)
• categories - Array of categories (replaced)
• people - Array of people names (replaced)

Response:

{
  "message": "Insight updated successfully",
  "id": "insight_xxx"
}

GET /api/insights/[id]?fetch=true - Get Insight

Get a single insight by ID, including associated chat.

curl "http://localhost:3414/api/insights/insight_xxx?fetch=true" \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "id": "insight_xxx",
  "chatId": "chat_xxx",
  "type": "decision",
  "content": "User decided to start new project next month",
  "chat": {
    "id": "chat_xxx",
    "title": "Chat with John",
    "messages": [...]
  },
  "createdAt": "2024-01-01T00:00:00Z"
}

DELETE /api/insights/[id] - Delete Insight

Delete a specific insight.

curl -X DELETE http://localhost:3414/api/insights/insight_xxx \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "success": true
}

GET /api/chat-insights?chatId=xxx - Get Chat Insights

Get all insights for a specific chat.

curl "http://localhost:3414/api/chat-insights?chatId=chat_xxx" \
  -H "Authorization: Bearer $TOKEN"

Insight Usage Analytics & Maintenance

openloomi tracks insight usage and performs periodic maintenance to preserve retrieval quality, avoiding context decay.

Usage Tracking

Each insight view is recorded with:

• Access timestamp
• Access source (list, detail, search, favorite)
• Cumulative access counts (7-day / 30-day / total)

Data is stored in the insightWeights table:

• accessCountTotal - Total access count
• accessCount7d - Access count in last 7 days
• accessCount30d - Access count in last 30 days
• lastAccessedAt - Last access timestamp

Analysis Dimensions

Each insight is scored on trend and value:

Trend Indicators:

• rising - Access frequency increasing
• falling - Access frequency decreasing
• stable - Frequency stable

Periodic Maintenance

System automatically runs two maintenance tasks:

Retention Policy:

• delete: 90 days no access + low score + not high importance → soft delete, hard delete after 180 days
• archive: 30 days no access + low score OR falling trend + low score → archived

GET /api/insights/analytics - Get Insight Usage Analytics

curl "http://localhost:3414/api/insights/analytics" \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "generatedAt": "2024-01-15T10:30:00Z",
  "summary": {
    "totalInsights": 150,
    "activeInsights": 45,
    "dormantInsights": 105,
    "totalAccesses30d": 230,
    "averageValueScore": 42,
    "risingInsights": 12,
    "fallingInsights": 8,
    "stableInsights": 25
  },
  "topInsights": [...],
  "bottomInsights": [...],
  "relationships": [...],
  "insights": [
    {
      "id": "insight_xxx",
      "title": "User decided to start new project",
      "description": "",
      "taskLabel": "",
      "platform": "gmail",
      "account": "user@gmail.com",
      "importance": "general",
      "urgency": "not_urgent",
      "isFavorited": false,
      "isArchived": false,
      "createdAt": "2024-01-01T00:00:00Z",
      "updatedAt": "2024-01-10T00:00:00Z",
      "time": "2024-01-01T00:00:00Z",
      "accessCountTotal": 15,
      "accessCount7d": 3,
      "accessCount30d": 8,
      "lastAccessedAt": "2024-01-15T10:30:00Z",
      "trend": "rising",
      "recent7dAccessCount": 3,
      "previous7dAccessCount": 1,
      "valueScore": 58,
      "recommendation": {
        "action": "keep",
        "reason": "Usage, freshness, or relevance still supports keeping it active."
      }
    }
  ]
}

POST /api/insights/[id]/view - Record Insight View

curl -X POST "http://localhost:3414/api/insights/insight_xxx/view" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"viewSource": "search"}'

Parameters:

• viewSource (string) - Source type: list, detail, search, favorite

Response:

{
  "ok": true
}

CLI Script

Quick Start

# Search ALL memory sources at once (recommended for comprehensive search)
node $SKILL_DIR/scripts/openloomi-memory.cjs search-all "query"

# Search local memory files (full-text, case-insensitive)
node $SKILL_DIR/scripts/openloomi-memory.cjs search-memory "boss"

# Search local memory files in specific subdirectory
node $SKILL_DIR/scripts/openloomi-memory.cjs search-memory "project" --directory=projects

# Search knowledge base (RAG, semantic search)
node $SKILL_DIR/scripts/openloomi-memory.cjs search-knowledge "project plan"

# List knowledge base documents
node $SKILL_DIR/scripts/openloomi-memory.cjs list-documents

# Get document content
node $SKILL_DIR/scripts/openloomi-memory.cjs get-document doc_xxx

# List recent insights (last 7 days)
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --days=7

# List insights from a specific channel (e.g., Gmail, Telegram, WhatsApp)
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=gmail --days=7
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=telegram --days=30
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=whatsapp

# Filter insights by keyword (supports multiple keywords - OR logic)
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --keyword=screen --keyword=linkedin --days=30

# Get single insight
node $SKILL_DIR/scripts/openloomi-memory.cjs get-insight insight_xxx

# Create a new insight
node $SKILL_DIR/scripts/openloomi-memory.cjs add-insight --title="Coffee preference" --description="I prefer Americano coffee" --importance=General

# Update an insight (partial update with array append)
node $SKILL_DIR/scripts/openloomi-memory.cjs update-insight insight_xxx --description="Updated description" --detail="User mentioned new preference"

# Delete insight
node $SKILL_DIR/scripts/openloomi-memory.cjs delete-insight insight_xxx

# Add a memory file
node $SKILL_DIR/scripts/openloomi-memory.cjs add-memory "My boss John likes Monday project discussions" --file=people/boss.md

# Delete a memory file
node $SKILL_DIR/scripts/openloomi-memory.cjs delete-memory people/boss.md

Command Reference

AI Agent Workflow

Triggered when the user asks about memory, knowledge, or past information:

1. Memory file search - "search my memory", "find what I said about..."
2. Knowledge base search - "search uploaded documents", "find in knowledge base"
3. Insights management - "list insights", "delete an insight"
4. Channel insights - "what messages on Gmail?", "show me Telegram chats", "any WhatsApp messages?"
5. Comprehensive search - "search everything", "find in all my memory", "build relationship graph"

Execution Flow:

1. Identify intent - determine if user wants comprehensive search or specific source
2. Prefer search-all - for general memory queries, always use search-all first to get comprehensive results across all sources
3. Execute in parallel - when specific sources are needed, run multiple searches simultaneously:
   • search-memory for local files
   • search-knowledge for uploaded documents
   • list-insights for extracted insights
4. For channel queries - use list-insights with --channel parameter:
   • "gmail" - Email messages via Gmail
   • "outlook" - Email messages via Outlook
   • "telegram" - Telegram chats
   • "whatsapp" - WhatsApp messages
   • "slack" - Slack messages
   • "discord" - Discord messages
   • "linkedin" - LinkedIn messages
   • "twitter" - Twitter/X posts
   • "weixin" - WeChat messages
   • "rss" - RSS feed items
5. Format output - aggregate and present results in user's language

Best Practice for Comprehensive Queries:

# When user asks about relationships, people, or general memory:
node $SKILL_DIR/scripts/openloomi-memory.cjs search-all "person/project/topic"

# Then optionally get details from specific sources
node $SKILL_DIR/scripts/openloomi-memory.cjs search-memory "person" --directory=people
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --days=30 --keyword=<keyword>

Channel-Based Message Queries:

# User asks "what emails did I receive?" or "show me Gmail messages"
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=gmail --days=7

# User asks "any Telegram messages about project X"?
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=telegram --days=30

# User asks "recent WhatsApp messages"?
node $SKILL_DIR/scripts/openloomi-memory.cjs list-insights --channel=whatsapp

Living Connections (Hebbian Potentiation)

Living Connections track relationships between insights that strengthen when they're accessed together. This implements Hebbian learning: "insights that fire together, wire together."

Commands

# Get related insights - "users who viewed X also viewed Y"
node $SKILL_DIR/scripts/openloomi-memory.cjs get-related-insights <insightId>

# Get related insights with filters
node $SKILL_DIR/scripts/openloomi-memory.cjs get-related-insights insight_xxx --limit=10 --minStrength=0.3

# Get connection statistics
node $SKILL_DIR/scripts/openloomi-memory.cjs get-connection-stats <insightId>

How It Works

1. When you view an insight, connections to other insights viewed within 5 minutes are strengthened
2. Connection strength decays over time using Ebbinghaus-style forgetting curve
3. Strong connections (strength > 0.5) are considered "living" - actively referenced

Response Format

{
  "insightId": "insight_xxx",
  "connections": [...],
  "relatedInsights": [
    {
      "insightId": "insight_yyy",
      "strength": 0.72,
      "coAccessCount": 5
    }
  ],
  "total": 5
}

Temporal Queries (Time-Travel)

Temporal validity enables "time-travel" queries - seeing what insights were relevant at a specific point in time.

Commands

# Get insights valid at a specific point in time (time-travel query)
node $SKILL_DIR/scripts/openloomi-memory.cjs get-insights-as-of 2026-01-01

# Get currently valid insights (no expiration or future expiration)
node $SKILL_DIR/scripts/openloomi-memory.cjs get-current-insights

# Get insights overlapping a time interval
node $SKILL_DIR/scripts/openloomi-memory.cjs get-insights-in-interval 2026-01-01 2026-06-01

Use Cases

• "What did I know about Project X on March 1st?"
• "What insights were valid during my vacation last July?"
• "Show me only currently relevant insights (hide expired ones)"

Entity Registry

Entity Registry tracks people, groups, concepts, projects, and companies as first-class entities with disambiguation support.

Commands

# List all entities of a specific type
node $SKILL_DIR/scripts/openloomi-memory.cjs list-entities --type=person

# Search entities by name
node $SKILL_DIR/scripts/openloomi-memory.cjs list-entities --search=John

# Get entity details with linked insights
node $SKILL_DIR/scripts/openloomi-memory.cjs get-entity <entityId> --insights

Entity Types

Search with Connections

Combined search that returns matching insights along with their Living Connections, providing a richer context.

# Search insights and include related insights
node $SKILL_DIR/scripts/openloomi-memory.cjs search-with-connections "project deadline"

# With custom limit
node $SKILL_DIR/scripts/openloomi-memory.cjs search-with-connections "project deadline" --limit=5

OpenLoomi Pet Sub-skill

The Loomi Pet has 9 universal state names. The plugin ships the fox (loomi-*) sprite set for branding; the OpenLoomi runtime's map_state_to_pet watcher renders the matching sprite for whichever theme you have active (fox or capybara, or any folder under ~/.openloomi/pet-custom/). State set:

Available commands

• node ${CLAUDE_PLUGIN_ROOT}/scripts/loomi-bridge.mjs pet <state> — synchronous, returns JSON; use only when the user explicitly asks.
• Hooks call state <name> automatically (fire-and-forget, 2s timeout).

Sprite set is hardcoded in the bridge — invalid state names are rejected before any HTTP call. The endpoint POST /api/pet/state may not yet exist in the target OpenLoomi runtime; the bridge falls back to "would have set state to X — pending OpenLoomi endpoint" without raising an error.

Help the user customize their pet's appearance

The bridge only drives the state — sprite overrides and theme folders are file-based and live outside the plugin. When the user asks to "change the pet's look" or "add a custom character", walk them through the file system. Do not try to write pet-config.json from the bridge; the bridge has no such command and the runtime's file watcher does the work.

Decision tree

1. "I just want the other built-in" → right-click Loomi → Theme → Fox / Theme → Capybara. Persisted in ~/.openloomi/pet-config.json under activeTheme. Long-press (~600 ms) is the fallback if contextmenu is swallowed by the host.
2. "I want my own character" → drop a folder at ~/.openloomi/pet-custom/<name>/ with PNGs named after the states. The watcher auto-discovers it within ~250 ms and the theme appears in the menu.
3. "I want to change just one sprite" → edit ~/.openloomi/pet-config.json's overrides map. Wins over both built-ins and custom-theme sprites for the matching (theme, state) pair.
4. "I want to make my theme the default" → set activeTheme to the custom theme's folder name.

What the bridge can and can't do

Filename conventions to communicate

When guiding the user through a custom theme, surface these conventions up-front:

• PNG only. .gif, .webp, .apng, .lottie are silently ignored.
• Bare or prefixed names both work. idle.png, loomi-idle.png, capybara-thinking.png, my-pack-sweeping.png all normalize correctly (case-insensitive).
• One recognizable state PNG is enough. The folder is registered as a theme as soon as it has ≥1 normalized state stem; missing states fall through to the active theme's idle sprite.
• Hidden / dot-prefixed folders are ignored. .git, .DS_Store etc.

Override JSON shape to communicate

The overrides map is camelCase on the wire — activeTheme, customThemesDir. Snake_case keys silently no-op the assignment (the unit test at apps/web/src-tauri/src/pet/theme.rs:499 pins the contract). Use the exact shape:

{
  "version": 1,
  "activeTheme": "fox",
  "customThemesDir": "~/.openloomi/pet-custom",
  "overrides": {
    "fox": {
      "idle": "/absolute/path/to/my-fox-idle.png"
    }
  }
}

Absolute paths only — the runtime routes them through tauri::convertFileSrc, so relative paths and ~/ prefixes will not resolve.

Common pitfalls to surface

• The folder doesn't show up — usually a missing or mis-named state PNG. Run them through the filename convention table.
• Override doesn't apply — host log line [loomi-pet/theme] failed to parse ~/.openloomi/pet-config.json means the JSON itself is malformed and defaults loaded. Host log line [loomi-pet/theme] failed to read <path>: <io error> means the path doesn't resolve.
• Theme menu tick is wrong / stuck — almost always a camelCase vs snake_case wire-format mismatch. The widget reads activeTheme, not active_theme.
• sleeping / sweeping rejected by /api/pet/state — those are watcher-only vocabulary; the runtime returns 400 invalid_state. Do not POST them from your tool.

What NOT to do from Claude

• Do not try to call POST /api/pet/state with sleeping or sweeping. The API rejects them; the watcher owns them.
• Do not try to write pet-config.json from the bridge. The bridge has no command for it; the file watcher owns updates.
• Do not invent new state names. CAPYBARA_STATES rejects them before any HTTP call.
• Do not claim support for non-PNG sprite formats. The asset pipeline is static PNG only.
• Do not say "I'll set up your custom theme for you" if you can only walk them through the file system. The watcher does the work; you guide.

See also

• Customize your Loomi Pet (user docs) — full guide to themes, custom folders, overrides, troubleshooting
• Pet API — POST /api/pet/state for external tools
• Attention Agent — the desktop pet as a whole
• The runtime source: apps/web/src-tauri/src/pet/theme.rs (custom themes + overrides), apps/web/src-tauri/src/pet/watcher.rs::map_state_to_pet (state resolution)

OpenLoomi Claude Plugin Skill

This skill is the single entry point for the OpenLoomi ↔ Claude Code integration. It is intentionally thin: it delegates to loomi-bridge.mjs for every side effect and never duplicates OpenLoomi business logic, connector implementations, or memory storage.

When to call loomi-bridge

Call loomi-bridge.mjs whenever ANY of the following is true:

• User says openloomi, loomi, pet, /openloomi:*, or asks about "the local AI assistant".
• User wants to install, configure, verify, or update OpenLoomi.
• User asks Claude Code to query their memory, look up a previous conversation, or "search my notes".
• User wants to set the Loomi Pet to a specific state.
• User wants today's LLM cost / usage.
• User asks about hooking Claude Code up to "the pet" or "loop".

When the user just wants something Claude can do natively (write a file, answer a question, etc.) and OpenLoomi is not mentioned, do not invoke the bridge.

Bridge subcommands (quick reference)

All subcommands emit JSON to stdout unless noted otherwise. All failure modes emit JSON (never bare stack traces).

Secrets contract (verbatim from plugins/codex/README.md §256–296)

Claude Code must never receive or print:

• model provider API keys (the plugin never reads them — the runtime handles its own AI provider configuration);
• OAuth access tokens or refresh tokens;
• connector app secrets;
• OpenLoomi auth tokens;
• local secure-storage contents (e.g. ~/.openloomi/token contents).

Allowed status-only checks:

OPENLOOMI_AUTH_TOKEN present/missing
~/.openloomi/token present/missing
native Claude CLI authenticated / not authenticated
AI provider configured/missing
connector configured/missing
local API reachable/unreachable

The bridge may report key names and presence. It must not print values. AI provider readiness comes entirely from the OpenLoomi runtime's /api/preferences/ai response (nativeRuntime / aiProviderConfigured).

Discovery chain

The bridge resolves your local OpenLoomi runtime in this order:

1. OPENLOOMI_BIN
2. OPENLOOMI_HOME / OPENLOOMI_INSTALL_DIR
3. OPENLOOMI_REPO_DIR
4. PATH lookup
5. Platform defaults — the desktop app's main binary:
   • macOS: ~/Applications/OpenLoomi.app/Contents/MacOS/openloomi
   • Linux: /opt/openloomi/openloomi (or ~/.local/bin/openloomi, /usr/local/bin/openloomi)
   • Windows: %LOCALAPPDATA%\OpenLoomi\openloomi.exe
6. ${CLAUDE_PLUGIN_DATA}/config.json (non-secret cached install path)
7. --bin-path <p> explicit flag
8. Otherwise: emit nextAction: install_openloomi

Hook events → Pet states

idle, sleeping, sweeping, presenting are managed by the loop watcher and are not set by hooks.

Reminders for Claude

• Never handle AI provider keys in this plugin. AI provider configuration lives in the OpenLoomi runtime; the runtime detects the user's local claude CLI auth on its own. If the user reports missing Claude CLI auth, point them at claude auth login or at OpenLoomi Desktop → API Settings for a custom endpoint.
• Never auto-install hooks. Always require an explicit /openloomi:hooks install.
• Always exit 0 on Stop. Archive failures are reported via stdout JSON with _openloomi.archive: "skipped", reason: ....
• When unsure, default to running loomi-bridge setup-status --json and respond based on the structured output — don't guess.

When to Apply

• User wants to access or interact with external apps (Gmail, Slack, GitHub, Notion, etc.)
• User wants to automate a task using an external service (send email, create issue, post message)
• Building an AI agent or app that integrates with external tools
• Multi-user apps that need per-user connections to external services

Setup

Check if the CLI is installed; if not, install it:

curl -fsSL https://composio.dev/install | bash

After installation, restart your terminal or source your shell config, then authenticate:

composio login       # OAuth; interactive org/project picker (use -y to skip)
composio whoami      # verify org_id, project_id, user_id

For agents without direct browser access: composio login --no-wait | jq to get URL/key, share URL with user, then composio login --key <cli_key> --no-wait once they complete login.

1. Use Apps via Composio CLI

Use this when: The user wants to take action on an external app directly — no code writing needed. The agent uses the CLI to search, connect, and execute tools on behalf of the user.

Key commands (new top-level aliases):

• composio search "<query>" — find tools by use case
• composio execute "<TOOL_SLUG>" -d '{...<input params>}' — execute a tool
• composio link [toolkit] — connect a user account to an app (agents: always use --no-wait for non-interactive mode)
• composio listen — listen for real-time trigger events

Typical workflow: search → link (if needed) → execute

Full reference: Composio CLI Guide

2. Building Apps and Agents with Composio

Use this when: Writing code — an AI agent, app, or backend service that integrates with external tools via the Composio SDK.

Run this first inside the project directory to set up the API key:

composio init

Full reference: Building with Composio

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi API Documentation

API Modules

All auth routes resolve against the local SQLite database. There is no cloud dependency — openloomi is fully self-contained. The remote-auth prefix is historical (the routes once proxied to a cloud server); today they are the canonical local endpoints, and the Claude/Codex plugin bridge uses /api/remote-auth/user as a port-discovery + auth-handshake probe.

This reference covers 131 route handlers under 36 top-level /api/* modules (auto-surveyed from apps/web/app/api/).

Functional Modules

Platform Callback Modules

Each integration platform has its own /api/<platform>/* module:

Endpoints Reference

Auth Module

Messages Module

Files Module

Storage Module

Integrations Module

Platform Callbacks

RAG Module

Workspace Module

AI Module

Chronicle Module

Insight Tabs Module

Chat Insights Module

Memory Module

Native Module

Pet Module

Loop Module (highlights)

24 routes total. Top-level surfaces:

Other Modules (single-route or paired)

Error Handling

Error Response Format

// API errors return standard HTTP status codes
{
  error: string;      // Error message
  code?: string;       // Error code for programmatic handling
  cause?: string;      // Additional context
}

Common Status Codes

AI/Agent Usage

Local API Access

When running openloomi desktop app, the local API server runs on port 3414 (fallback: 3515):

Authentication Token

The auth token is stored at ~/.openloomi/token (base64 encoded JWT). You must decode it before use:

# Decode base64 to get JWT token
TOKEN=$(cat ~/.openloomi/token | base64 -d)

# Verify token contents (decodes JWT payload)
echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | python3 -m json.tool

curl Examples

Important: All authenticated requests require the token to be base64 decoded first.

# Helper: Get decoded token
TOKEN=$(cat ~/.openloomi/token | base64 -d)

# 1. Check AI API status (no auth required)
curl http://localhost:3414/api/ai/chat

# 2. Get current user info (also used by Claude/Codex plugin as a port-discovery probe)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl http://localhost:3414/api/remote-auth/user \
  -H "Authorization: Bearer $TOKEN"

# 3. Create an anonymous guest session (no credentials)
curl -X POST http://localhost:3414/api/remote-auth/guest \
  -H "Content-Type: application/json" \
  -d '{}'

# 4. Chat with AI (streaming)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/ai/chat \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"stream":true}'

# 5. Get chat insights (requires chatId)
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl "http://localhost:3414/api/chat-insights?chatId=xxx" \
  -H "Authorization: Bearer $TOKEN"

# 6. Search RAG documents
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/rag/search \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"search term","limit":5}'

# 7. List workspace skills
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl http://localhost:3414/api/workspace/skills \
  -H "Authorization: Bearer $TOKEN"

# 8. Submit feedback
TOKEN=$(cat ~/.openloomi/token | base64 -d)
curl -X POST http://localhost:3414/api/remote-feedback \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"content":"Feedback message","email":"user@example.com"}'

Summary

• 131 route handlers across 22 functional modules + 12 platform callback modules + 2 cross-cutting modules (proxy, db)
• Fully self-contained: all auth, data, AI, and sync run locally — no cloud dependency
• Dual authentication: Session cookies (web) and Bearer tokens (Tauri)
• RESTful JSON APIs with Zod validation
• SWR utilities for client-side data fetching
• OAuth support for Slack, Discord, X, HubSpot, LinkedIn, Notion
• RAG for document retrieval and search
• AI endpoints for chat, images, audio
• Loop for attention loop, decisions, channels, classifier rules
• Pet state mirror (read/write /api/pet/state)

OpenLoomi Connectors

Use this skill as a thin wrapper for connector readiness guidance. Do not implement connector protocols in Codex and do not ask users to paste OAuth tokens, API keys, bot tokens, cookies, or connector secrets into Codex chat.

First, load workflow guidance:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-connectors

Then check readiness:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status

Report connector state as status and next action only. If a connector is missing or unavailable, guide the user to OpenLoomi-owned setup surfaces. If the runtime is ready and the user asks for a connector-backed task, pass the request over stdin:

printf "%s" "<user connector request>" | node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" run

Keep connector authentication, sync, message access, and platform-specific actions inside OpenLoomi runtime.

When setup-status includes connectorStatusAvailable: true, report only the status-only connector fields such as id, connected, and accountCount. When it includes connectorSetupRecommended: true, treat recommendedNextAction: "configure_connectors" as a non-blocking setup recommendation and hand the user to the reported OpenLoomi /connectors URL. Do not treat this as core runtime failure when ready: true.

setup-status may merge Loop connector rows with OpenLoomi native integration accounts (for example QQbot or Feishu) as status-only connector rows. Report that connected state when present, but keep all authentication, sync, and account management inside OpenLoomi.

For accounts connected through Composio (a broader 1000+ apps surface), treat the composio skill as a sibling status-only source. When the user asks "what am I connected to?" or "list all linked accounts", invoke the composio skill in parallel (composio-cli list-connections, or mcp__composio__COMPOSIO_MANAGE_CONNECTIONS with action: "list") and merge its results with the setup-status connector rows before reporting. Keep authentication, OAuth, and disconnect flows native to each skill — do not route Composio auth or connection management through OpenLoomi.

Note: If you haven't downloaded or installed openloomi yet, please refer to Getting Started for installation instructions.

OpenLoomi Product Features

Use this skill when users ask about openloomi features, usage, or capabilities. Provide accurate and easy-to-understand feature introductions and operation guides.

What is OpenLoomi

openloomi is a Proactive AI Workspace that understands your intent, orchestrates execution, and gets things done. It's not just another AI assistant—it's an innovative AI product that senses business signals, orchestrates tasks autonomously, and tracks and validates results end-to-end.

Core Value Proposition

openloomi transforms how individuals and SMB teams work by:

• Proactive Awareness — Monitors signals across platforms and alerts you before you ask
• Long-Term Memory — Remembers context across months, never forgets commitments
• Autonomous Execution — Not just telling you what to do, but doing it
• Builtin Skills — Rich execution capabilities for every work scenario

Core Capabilities

🧠 Long-Term Memory

Clear recollection, never forgotten. openloomi builds persistent knowledge graphs that remember all important people, events, decisions, and context across sessions and time. Six months later, it still knows your commitments and preferences.

🎯 Noise Filtering

Tells you what you should act on. With hundreds of daily messages, openloomi replaces "information overload" with "priority signals." Filters 95% of noise, focusing your attention on the 5% that truly matters.

⚡ Powerful Engine

Intent understanding, automatic orchestration. When you say "Help me prepare an investor pitch," openloomi automatically understands intent, breaks it into multiple sub-tasks, invokes appropriate Skills, and chains execution.

🔐 Security & Privacy

Your data, your sovereignty. Local-first architecture—your raw data never leaves your device. End-to-end AES-256 encryption, zero-data-training commitment, SOC 2 compliance audit.

🛠️ Builtin Skills

Builtin Skills covering every work scenario, continuously expanding:

• 📊 Data Analysis
• 💻 Code Generation
• 📄 Document Creation
• 🌐 Web Automation
• 🎨 Image Generation
• 📧 Email Writing
• 🔍 Deep Research
• 📊 PPT Creation
• more skills...

Use Cases

🌍 Global Managers

Never miss a critical signal across time zones. Business runs 24/7 globally. openloomi filters time zone and language noise, capturing high-value opportunities while you sleep—wake up to a refined action list.

🧑‍💻 Engineers & Product Teams

Team memory that never decays. Transform discussions scattered across Slack, Jira, and documents into structured knowledge. Auto-generate weekly reports, sync missed context, eliminate "context rot."

🚀 Founders & Sales

One person does the work of many, at scale. openloomi learns your communication style, automatically maintains hundreds of client relationships, follows up on leads, generates personalized proposals—never burns out.

Quick Start

1. Sign Up / Sign In

• Sign up with email and password
• Or sign in directly with your Google or GitHub account

2. Onboarding

First-time users will go through an onboarding flow:

1. Select your role and focus areas
2. Tell openloomi what you'd like it to help with
3. Connect platforms to unlock deeper insights
4. Name your AI assistant

3. Connect Communication Platforms

Click [Connect platform] to complete authorization.

Supported platforms:

• Messaging: Slack, Telegram, Discord, WhatsApp, Weixin, iMessage, QQ, Feishu, DingTalk
• Email: Gmail, Outlook
• Social Media: X (Twitter) — for marketing and content automation
• Other: RSS
• Coming Soon: Google Drive, Microsoft Teams, Notion, HubSpot, Google Calendar

Platform Connection Steps

WhatsApp

1. Click [Connect WhatsApp]
2. Complete authorization via QR code scan or phone pair code
3. Once authorized, openloomi will automatically read your WhatsApp messages to generate long-term events. You can send messages in "Starred Messages" and AI will:

• Read and understand your message content
• Generate smart insights in openloomi
• You can converse with AI about these messages in openloomi

💡 How to use: Open "Starred Messages" in WhatsApp, send a message to yourself, and AI will automatically read and understand it.

Weixin

1. Click [Connect Weixin]
2. Complete authorization via QR code scan

Telegram

1. Click [Connect Telegram] to enter the authorization page in the source settings.
2. Choose a login method:
   • Phone verification: Enter phone number → receive verification code → enter 2FA password if enabled
   • QR code: Scan QR code with Telegram → enter 2FA password if enabled
   • Quick login: If you have the official Telegram desktop app installed locally, you can use your existing session to log in without phone number or verification code
3. Once authorized, openloomi will automatically read your Telegram messages to generate long-term events on the Today page. You can send messages in "Saved Messages" and AI will:

• Read and understand your message content
• Generate smart insights in openloomi
• You can converse with AI about these messages in openloomi

💡 How to use: Open "Saved Messages" in Telegram, send a message to your saved messages, and AI will automatically read and understand it.

Slack

1. Click [Connect Slack] in the integration settings
2. Click [Install openloomi] on the authorization page to add to your workspace
3. Note: Currently only workspace owners can install

Discord

1. Click [Connect Discord] in the integration settings
2. Select the Discord server to install
3. Grant openloomi bot message permissions
4. Note: Only server admins can install

Gmail

1. Click [Connect Gmail] in the integration settings
2. Enter the email address and app password to authorize

RSS

1. Click [RSS] button to enter the RSS integration page
2. Enter a single RSS link, or upload an OPML file for batch import

Feishu

1. Click [Connect Feishu]
2. Enter your Feishu App ID and App Secret
3. Click connect

How to get credentials:

1. Go to Feishu Open Platform
2. Create an enterprise self-built app
3. Enable bot capability
4. Select "Use long connection to receive events"
5. Subscribe to im.message.receive_v1
6. Get App ID and App Secret from the app settings

Required permissions (scopes):

{
  "scopes": {
    "tenant": [
      "aily:file:read",
      "aily:file:write",
      "application:application.app_message_stats.overview:readonly",
      "application:application:self_manage",
      "application:bot.menu:write",
      "cardkit:card:write",
      "contact:user.employee_id:readonly",
      "corehr:file:download",
      "docs:document.content:read",
      "event:ip_list",
      "im:chat",
      "im:chat.access_event.bot_p2p_chat:read",
      "im:chat.members:bot_access",
      "im:message",
      "im:message.group_at_msg:readonly",
      "im:message.group_msg",
      "im:message.p2p_msg:readonly",
      "im:message:readonly",
      "im:message:send_as_bot",
      "im:resource",
      "sheets:spreadsheet",
      "wiki:wiki:readonly"
    ],
    "user": [
      "aily:file:read",
      "aily:file:write",
      "im:chat.access_event.bot_p2p_chat:read",
      "im:chat:read",
      "im:chat:readonly"
    ]
  }
}

X (Twitter)

Connect X (Twitter) to enable marketing automation features.

1. Click [Connect X] to authorize via OAuth

Outlook

1. Click [Connect Gmail] in the integration settings
2. Enter the email address and app password to authorize

Microsoft Teams

Coming soon!

QQBot

1. Click [Connect QQ]
2. Enter your QQ App ID and App Secret
3. Click connect

How to get credentials:

1. Go to QQ Open Platform
2. Create a bot
3. Get App ID and App Secret from the bot settings

DingTalk

DingTalk integration uses a Stream mode bot — a long-lived WebSocket connection with no public IP or domain required.

Before you start — create a DingTalk app:

1. Go to DingTalk Open Platform and sign in
2. Create an enterprise internal app
3. Add the Bot capability and choose Stream mode (long connection)
4. Copy your Client ID (AppKey) and Client Secret (AppSecret)

Connect in openloomi:

1. Click [Connect DingTalk]
2. Enter your Client ID (AppKey) and Client Secret (AppSecret)
3. Click Connect

💡 Stream mode means openloomi connects directly via WebSocket — no server or domain setup needed.

iMessage

iMessage integration is only available on macOS.

1. Click [Connect iMessage]
2. Grant the required permissions:
   • Full Disk Access - Required to read iMessage database
   • Automation Permission - Required to send messages
3. Enter a display name for your iMessage account
4. Click connect

How to grant permissions:

• Go to System Settings > Privacy & Security > Full Disk Access
• Add the running app (Terminal, Node, or openloomi)
• Restart the app after granting permissions

💡 Your message data stays on your local device. openloomi only reads recent messages when you use it to generate insights.

Desktop App

openloomi also offers a desktop app (macOS, Linux, and Windows) that provides a native local experience.

Download & Install

Windows:

1. Download the latest .exe installer from GitHub Releases
2. Run the installer — if Windows SmartScreen shows a warning, click "More info" then "Run anyway". This is normal for new applications without a long code-signing reputation; openloomi is open-source and the code is publicly verifiable
3. After the first run, SmartScreen typically bypasses automatically on subsequent updates

Winget (Coming Soon):

winget install openloomi.openloomi

💡 You only need to complete the SmartScreen step once per machine.

macOS:

Download the latest .dmg from GitHub Releases and drag openloomi to your Applications folder.

Linux:

Download the latest .AppImage or .deb from GitHub Releases.

Important: App Must Be Running

To use MessageApp conversations and scheduled automation tasks in the desktop app:

• The app must be open and running
• The computer must be turned on and not in sleep mode

If the desktop app is closed or the computer is asleep, conversations and scheduled tasks will not execute.

Local Data Storage

All data in the desktop app — including messages, conversations, scheduled tasks, and settings — is stored locally on your device via SQLite. No app data is sent to or stored on cloud servers.

Permissions

When you first launch openloomi, the system may ask for a few permissions. Each one has a specific purpose — and you can decline any of them. openloomi will continue to work; you'll just lose the feature that requires that permission.

macOS

How to grant or revoke:

1. Open System Settings → Privacy & Security
2. Find the permission category (e.g., Full Disk Access, Automation, Notifications)
3. Toggle openloomi on or off

You can revisit these settings at any time.

Windows

Windows SmartScreen may also show a one-time warning when running the installer. Click "More info" then "Run anyway" — this is normal for open-source software. You only need to do this once per machine.

Linux

No special permissions are required on Linux. openloomi uses the standard desktop notification system (libnotify) to send alerts — if you have granted notification permissions to other apps, openloomi will use them automatically.

Conversation Features

Just type your questions or requests in the chat box, and openloomi will help you find answers.

Example Questions

• "What is openloomi"
• "How to use openloomi"
• "Summarize yesterday's to-dos"
• "Today's important news"
• "What are my contacts"
• "Randomly send 'Hello' to 3 contacts on Gmail"
• "What progress have we made with the XX project this past week?"

Features

• Project collaboration queries - Ask about project progress
• Weekly report generation - Request weekly reports on all project progress
• Web browsing - Have openloomi browse for latest product info
• New conversation creation - Start new conversations around specific topics
• History - View conversation history
• Source References - See exactly which messages/conversations openloomi's answers come from, who was involved, and when they occurred
• Artifacts - openloomi can generate visual artifacts: mind maps, flowcharts, charts, roadmaps, surveys, and documents. Preview and interact with them directly in chat
• File Analysis - Upload files (PDF, images, etc.) in chat and ask openloomi to analyze, summarize, or extract information
• Deep Dive - For certain topics, continue exploring with follow-up questions, detail requests, or scope narrowing
• Topic-Based Chats - Create new conversations around specific topics, review past discussions, build persistent context over time

Chat via Messaging Apps

You can also interact with openloomi directly through your connected messaging apps. Once connected, openloomi becomes your AI assistant within those platforms.

After connecting (e.g., Telegram), just send a message to Saved Messages and openloomi will respond naturally.

Action Features

openloomi can generate action items for various scenarios.

How to View

1. Click the Action button in the event details
2. Or view in the unified Action panel

Features

• To-do display - Show TODOs in the unified panel
• Quick action suggestions - openloomi suggests clickable quick actions
• Detailed information - Fill in sender, recipient, content and attachments
• AI content generation/translation - Auto-generate or translate content
• Message replies - Click reply button in understanding detail view, openloomi will generate a reply

Smart Insights

openloomi automatically analyzes your conversations to extract valuable information.

Automatically Extracted Content

• ✅ To-dos - Tasks to complete, deadlines
• 📈 Project progress - Status updates, milestones
• 🎯 Important decisions - Meeting decisions, key choices
• ⚠️ Risk alerts - Issues to watch out for
• 📅 Timeline - Event development脉络

How to Use

1. View - Click on an event in the left menu
2. Categorize - Mark as: Urgent, Important, Monitor, Archive
3. Add to-do - Add tasks directly in the details
4. Timeline - View event development timeline

Event Management

Events are automatically organized into groups based on their status:

• Opportunities - New events that need attention
• In Progress - Events you're currently working on
• Waiting on Others - Events pending response
• Done - Completed events

You can drag and drop events between groups to manually categorize them. You can also multi-select multiple events for bulk operations (Mark as Done, Archive, Delete).

Event Detail

Click on any event card to open its detail view:

• Event title and description - Full context of the event
• Related messages - All conversation threads involved
• Participants - Everyone included in the event
• Timeline - Chronological activity log
• Notes - Add personal text notes for reference
• Attachments - Upload documents, PDFs, images to keep related materials in one place

Event Conversation

You can chat directly with any event to ask questions and get AI insights within that context.

Sample questions:

• "What is this event about?"
• "Who are the key people involved?"
• "What decisions were made?"

Event Actions

• Source Reply - Reply directly to messages within an event. Use AI suggestions to generate a reply based on conversation context, or use AI Translation to translate to English or Chinese
• AI Polishing - Improve grammar, wording, and tone (formal, casual, friendly)
• Send to Platform - Send your reply directly to the original platform (Telegram, Discord, etc.)

Common Queries

"What's on my to-do list today?"
"How is the XX project progressing?"
"What important messages were there last week?"

Settings

How to access: Click the settings button in the profile panel

Soul

Define your AI assistant's personality and communication style.

Description

Customize how your AI assistant describes itself to others.

Contexts

Configure which data sources and contexts your AI assistant can access. Context types include:

• System - System notifications and status updates
• Event - Grouped communications and projects
• Scheduled Task - Time-based tasks and reminders
• Knowledge - Uploaded documents and reference materials

You can also create custom context tabs with your own name, description, icon, color, and priority. Use keywords to enable automatic categorization.

Interests

Customize what to follow — specific people or topics/projects. For each, you can set:

• Notification Level: All messages, Only @me, or Nothing
• AI Summary: Enable AI-generated summaries of their messages
• Auto-archive: Automatically archive related messages

Connectors

Manage all your connected platforms and services in one place.

Disconnect / Revoke Access

If you no longer want openloomi to access a connected platform, you can disconnect it at any time:

1. Go to Settings → Connectors
2. Find the platform you want to disconnect
3. Click [Disconnect] or the remove (×) button
4. Confirm the action

Once disconnected:

• openloomi will immediately stop reading new messages from that platform
• Previously synced data is retained until manually deleted
• You can reconnect at any time by repeating the connection steps

💡 Tip: Before disconnecting, you may want to review what data has been synced in the Privacy & Security settings.

Language

Change the language used in openloomi.

Search

Search across all your messages, files, and conversations to find exactly what you need.

Scheduled Tasks

Have AI automatically execute tasks at specified times.

How to Create

1. Go to Agent/Automation page
2. Click "New Task"
3. Fill in task information:

Schedule Types

• Cron Expression — Flexible scheduling with cron syntax (e.g., 0 9 * * * = every day at 9am, 0 9 * * 1 = every Monday at 9am)
• Interval — Run every X minutes/hours (e.g., every 30 minutes, every 2 hours)
• One-Time — Run once at a specific date and time

Manage Tasks

• Enable/Disable — Turn tasks on or off
• Run Now — Execute immediately without waiting for the scheduled time
• View History — See past execution results, success/failure status, and output logs
• Edit — Modify task configuration
• Delete — Remove a task

Example Use Cases

• Daily News Summary: "Search latest AI news, summarize top 5 stories and email to me" — every morning at 8am
• Weekly Report: "Generate weekly report on all project progress" — every Friday at 5pm
• Periodic Reminder: "Check calendar for upcoming meetings, remind me 15 minutes before" — every 30 minutes

Knowledge Base

After uploading documents, you can ask AI questions about them directly.

How to Use

1. Upload documents - Upload PDF, Word, text, etc. in settings
2. Ask questions - Ask AI "What's in the document about XXX?"
3. Get full content - View the complete document when needed

Privacy Policy

Your privacy matters. Our Privacy Policy explains in detail how we collect, use, store, and protect your data — including what data we access, how it's encrypted, how long we retain it, and your rights to access, export, or delete it at any time.

Privacy & Security

Your data, your sovereignty. openloomi puts privacy and control first—you never need to trade data sovereignty for intelligence.

🔐 Our Privacy Principles

Local-First Architecture

Your original messages and files stay on your device. openloomi only accesses the minimum data needed to complete tasks—no unnecessary uploads.

• Raw data never leaves your local environment
• Only processed results are transmitted when needed
• Complete control over what data openloomi can access

End-to-End Encryption

All authorized data is encrypted with AES-256 industry standard, processed in hardware-isolated trusted execution environments.

• AES-256 encryption for all stored data
• Hardware-isolated trusted execution environments
• Zero visibility into your raw data

Zero Training Commitment

Your data will never be used to train public AI models—now or ever. In writing, legally guaranteed.

• Explicit legal commitment in our terms of service
• No use of your data for model training
• Full transparency on data usage

Auditable Compliance

All critical operations are logged in tamper-proof audit logs. SOC 2 compliance certification in progress.

• Complete audit trail of all data access
• SOC 2 compliance certification
• Enterprise-grade security standards

🔑 Your Data, Your Control

openloomi gives you complete control over your data:

• Access Controls: Choose exactly which platforms and data openloomi can access
• Data Retention: Configure how long openloomi retains processed data
• Delete Anytime: Delete all your data at any time with one click
• Export: Export all your data in standard formats

We believe privacy and intelligence aren't mutually exclusive. You should never have to choose between powerful AI capabilities and data control. openloomi delivers both.

FAQ

Q: How do I use openloomi?

A: It's simple!

1. Start a conversation in openloomi
2. Describe your needs directly
3. AI will understand and execute

Q: How do I connect Telegram?

A:

1. Click [Connect Telegram]
2. Choose phone verification or QR code login

Q: What are Insights?

A: Automatically extracted important information from your conversations, including to-dos, project progress, decision records, etc.

Q: How do I switch the desktop app to use the Codex CLI as its agent runtime?

A: When OpenLoomi is used from Codex, the Codex runtime is the recommended desktop runtime. It lets OpenLoomi reuse the user's existing Codex CLI runtime instead of requiring a separate OpenLoomi AI provider key for the first plugin workflow.

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" codex-runtime-info

Follow the returned platform-specific guidance, restart OpenLoomi, then verify the active provider through /api/native/providers.

Q: How do I create automation tasks?

A:

1. Go to Agent/Automation page
2. Click "New Task"
3. Fill in task name and description
4. Set execution time
5. Save

Showcase: openloomi in Action

Real-world examples showing how openloomi works across different use cases.

Chat Scenarios

Office: Smart Meeting Coordination

Scenario: A team needs to schedule a cross-timezone meeting.

User: Hey, find a time this week when everyone is available in the afternoon

openloomi:
📅 Found available slots for 5 team members:

| Time (PT) | Wed | Thu | Fri |
|-----------|-----|-----|-----|
| 2-3pm     | ✅  | ✅  | ❌ |
| 3-4pm     | ✅  | ✅  | ✅ |
| 4-5pm     | ❌  | ✅  | ✅ |

Recommended: Thursday 3-4pm (everyone available)
→ Want me to create a Google Calendar meeting?

What it shows: Google Calendar integration, multi-account availability lookup, natural language scheduling

R&D: Code Review Context

Scenario: A developer gets PR feedback and needs full context.

User: What's the status of PR #247?

openloomi:
📋 PR #247 Review Summary:

🔍 Related Discussion:
- @john: Suggested refactoring auth logic (comment #3)
- @sarah: Requested additional unit tests
- @mike: Approved with "LGTM"

📎 Linked Slack Threads:
#engineering: "Auth module refactoring plan" (3 messages)

📝 Extracted Tasks:
- [ ] Add auth test cases
- [ ] Update API documentation
- Due: This Friday

→ Want me to generate suggested changes?

What it shows: Cross-platform context aggregation, task extraction, timeline tracking

Marketing: Campaign Coordination

Scenario: Marketing team coordinating a product launch.

User: Is our Wednesday product launch ready?

openloomi:
🚀 Product Launch Status:

| Item | Status | Owner |
|------|--------|-------|
| Website update | ✅ Done | @marketing |
| Social posts | 🔄 In Progress | @social |
| Email blast | ⏳ Pending | @email |
| KOL outreach | ❌ Blocked | @partnerships |

⚠️ Blockers:
- KOL partnership awaiting legal approval
- Waiting for final product documentation

→ Want me to remind the team?

What it shows: Task status tracking, cross-team coordination, blocker identification

Collaboration: Project Kickoff

Scenario: Starting a new project with rapid setup.

User: Create a new client project called Acme Corp

openloomi:
✅ Project Acme Corp created!

Auto-completed:
📁 Google Drive folders
  - /Acme Corp/Contracts
  - /Acme Corp/Meeting Notes
  - /Acme Corp/Requirements

💬 Slack channels
  - #acme-general (project overview)
  - #acme-dev (technical)
  - #acme-support (client support)

📅 Shared calendar
  - Kickoff meeting (tomorrow 10am)
  - Weekly standup (Fridays 3pm)

👥 Team invitations sent

What it shows: Multi-platform automation, workflow orchestration

Tracking Scenarios

Proactive: High-Priority Alerts

Scenario: System proactively identifies items needing attention.

📊 Today's Insights - March 12, 2026

🔥 Urgent (3)
├── PR #251 needs review - waiting 4 hours
├── Support tickets piling up - 12 unprocessed
└── Server alert - CPU at 95%

⭐ Important (5)
├── Q1 budget approval - due tomorrow
├── Acme contract renewal - 30 days left
└── ...

📋 Todo Tasks (8)
├── Reply to John's technical question - due today
├── Approve 3 team PTO requests
└── ...

→ View All | Mark Read | Remind Later

What it shows: Importance/urgency ranking, unreplied tracking, deadline reminders

Auto-Managed: Scheduled Refresh

Scenario: System automatically keeps information up to date.

⏰ Scheduled jobs running...

✅ RSS Feeds Updated
├── Tech News (5 new articles)
├── Product Management (2 new articles)
└── Competitor Updates (8 new articles)

✅ Insights Refreshed
├── Refresh interval: 30 minutes (high activity)
├── New insights: 6
└── Archived: 2

✅ Email Automation Triggered
├── User signed up 7 days ago → onboarding email
├── Free quota running low → upgrade reminder
└── Inactive 30 days → re-engagement email

What it shows: Scheduled RSS fetching, automated insight refresh, user lifecycle automation

IM Platform Scenarios

Feishu: Enterprise Workflow

Scenario: Enterprise workflow automation.

Feishu Group:
User: Requesting PTO March 20-22

openloomi:
📝 PTO Request

| Field | Content |
|-------|---------|
| Applicant | @john |
| Dates | March 20-22 (3 days) |
| Type | Annual Leave |
| End Date | N/A |

Approver: @manager

[Approve] [Reject] [Request Info]

What it shows: Approval workflows, message handling, document integration

WhatsApp: Personal Assistant

Scenario: Managing personal tasks via WhatsApp.

WhatsApp conversation:
User: Set a reminder for tomorrow at 2pm

openloomi: ✅ Reminder set

⏰ Tomorrow 2:00pm
📌 Reminder: Call with John
📋 Context: Discussing Q1 proposal
   Client: john@acme.com
   Budget: $50,000

What it shows: QR pairing, status polling, reminder management

Discord: Community Management

Scenario: Automated community operations.

New member joins Discord server

openloomi Welcome Bot:
👋 Welcome @newuser!

I'm openloomi, your community assistant:

📚 Resources
- #announcements - Latest updates
- #help - Get support
- #showcase - Share projects

⚡ Quick Start
- [Introduce Yourself] Share your background
- [Set Up Profile] Customize your info
- [Browse Channels] Explore the community

Ask me anything!

What it shows: New member onboarding, FAQ automation, message handling

🚀 Industry Intelligence Automation

Scenario: Automated AI product news monitoring with daily briefings.

User: Help me set up an industry intelligence automation

openloomi:
✅ Setting up your AI Product Daily Digest Assistant!

Step 1/4: Create Scheduled Task
────────────────────────────────
Task Name: AI Product Daily Digest
Schedule: Daily at 9 AM (Cron: 0 9 * * *)
Description:
1. Scrape X, Reddit, Product Hunt for AI product news
2. Summarize top 10 stories
3. Record to timeline as "AI Product Daily Updates"
4. Send briefing via Telegram

Step 2/4: Connect Data Sources
────────────────────────────────
✅ X (Twitter) - Connected
✅ Reddit - Connected
✅ Product Hunt - Connected
✅ Telegram - Connected

Step 3/4: Task Created
────────────────────────────────
🎉 Your automation is now active!

Next run: Tomorrow at 9:00 AM

Step 4/4: What You'll Receive
────────────────────────────────
☀️ Daily Telegram Briefing:
• Top AI product launches
• Trending discussions
• Engagement metrics

📰 Timeline Event:
• "AI 产品要闻每日更新"
• Full context for follow-up questions

🔮 Coming Soon:
• Visual dashboards
• Team sharing to Slack

What it shows: Scheduled tasks, multi-platform scraping, automated briefings, timeline recording

Example: Daily Briefing Output

☀️ AI Product Daily Digest - March 15, 2026

🔥 Top 5 AI Product Launches Today:

1. 🎨 Claude Art (Product Hunt)
   AI image generation with style transfer
   247 upvotes

2. 💻 Devin 2.0 (X)
   AI coding assistant v2.0
   1.2K retweets

3. 🔧 LangChain v1.0 (Reddit)
   Major agent framework update
   89 upvotes

📈 Trend Summary:
- Image Generation: 🔥 Hot
- AI Coding: 📈 Growing

[View Full] [Create Follow-up] [Share]

What it shows: Multi-source aggregation, smart summarization, actionable outputs

Reference

• openloomi website: https://openloomi.ai
• openloomi documents: https://openloomi.ai/docs

OpenLoomi Handoff

Use this skill as a thin wrapper for OpenLoomi handoff workflows. Do not build a separate task queue, reminder store, or persistence layer inside the Codex plugin.

First, load workflow guidance:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-handoff

Then check readiness:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status

If ready: false, follow the reported nextAction before attempting handoff. When ready: true, wrap the handoff request with the taskPromptPrefix returned by workflow-guidance, then pass that runtime-safe prompt over stdin:

printf "%s" "<taskPromptPrefix>\n\nOriginal user request: <handoff request>" | node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" run

Do not send wording that asks the inner runtime to invoke Codex plugins, OpenLoomi plugins, skills, shell commands, or loomi-bridge. Include enough task context for OpenLoomi to create a follow-up, but do not include secrets. OpenLoomi runtime owns handoff persistence and notification routing.

OpenLoomi Install Sub-skill

This sub-skill is auto-loaded when the user wants to install or fix OpenLoomi on their machine. It composes loomi-bridge operations and never downloads or executes anything outside the plugin's own scripts.

Quick workflow

1. node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status
2. Based on nextAction / reason:

Codex runtime setup

If the user is setting up OpenLoomi from Codex, explicitly asks to switch the desktop runtime executor to Codex, or diagnostics show a native-agent provider mismatch, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" codex-runtime-info

Reminder: secrets contract

• The bridge never reads AI provider env vars. Provider readiness comes from the OpenLoomi runtime's /api/preferences/ai. The bridge only sees presence booleans (configured / not configured), never key values.
• If the user pastes a key into chat, redact it: do NOT echo it back, and tell them to remove it from chat history.
• Never pass --api-key as an argv flag. The bridge has no such flag, by design.
• The bridge does not auto-install. Always require an explicit user confirmation before calling install-openloomi --confirm.

OpenLoomi Loop

Use this skill as a thin wrapper for OpenLoomi loop workflows. Do not implement loop scheduling, decision storage, connector checks, or memory logic in Codex. OpenLoomi runtime owns those behaviors.

First, load workflow guidance:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-loop

Then check readiness:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status

If ready: false, follow the reported nextAction and do not continue the loop task yet. Guest/session initialization must happen through OpenLoomi-owned surfaces. Never ask for API keys, OAuth tokens, connector secrets, or OpenLoomi auth tokens in Codex chat.

When ready: true, wrap the user request with the taskPromptPrefix returned by workflow-guidance, then pass that runtime-safe prompt over stdin to the bridge:

printf "%s" "<taskPromptPrefix>\n\nOriginal user request: <user loop request>" | node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" run

Do not send wording that asks the inner runtime to invoke Codex plugins, OpenLoomi plugins, skills, shell commands, or loomi-bridge. Keep all persistence, connector state, memory access, and follow-up scheduling inside OpenLoomi runtime.

OpenLoomi Memory

Use this skill as a thin wrapper for OpenLoomi memory workflows. Do not read or write OpenLoomi memory files directly from Codex, and do not copy memory implementation details into this plugin.

First, load workflow guidance:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-memory

Then check readiness:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status

If ready: false, follow the reported nextAction. Connector setup and guest/session initialization must happen through OpenLoomi-owned surfaces, not Codex chat.

When ready: true, wrap the user request with the taskPromptPrefix returned by workflow-guidance, then pass that runtime-safe prompt over stdin to the bridge:

printf "%s" "<taskPromptPrefix>\n\nOriginal user request: <user memory request>" | node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" run

Do not send wording that asks the inner runtime to invoke Codex plugins, OpenLoomi plugins, skills, shell commands, or loomi-bridge. Only show memory content when OpenLoomi runtime returns it for the requested task. Keep secrets and connector credentials out of prompts, argv, stdout, and stderr.

OpenLoomi Pet Sub-skill (Codex)

Codex parity note: this skill ships only in the Claude plugin today. The Codex plugin's loomi-bridge.mjs already exposes the same pet <state> command and lifecycle hooks (see plugins/codex/README.md § Codex Pet lifecycle hooks), and the file-based theme system is runtime-side, identical across both plugins. The Codex plugin does not yet have its own openloomi-pet SKILL.md — this stub mirrors the Claude one so users on either surface get the same guidance.

When Codex's pet <state> behaviour diverges from Claude's, this file is the authoritative Codex-side reference.

The Loomi Pet has 9 universal state names. The runtime's map_state_to_pet watcher renders the matching sprite for whichever theme is active (fox, capybara, or any folder under ~/.openloomi/pet-custom/). State set:

Available commands

• node ${CODEX_PLUGIN_ROOT}/scripts/loomi-bridge.mjs pet <state> — synchronous, returns JSON; use only when the user explicitly asks.
• Hooks call state <name> --event <event> automatically (fire-and-forget, 2s timeout).

The Codex bridge mirrors the Claude bridge's cmdPet. Invalid state names are rejected client-side before any HTTP call. The endpoint POST /api/pet/state may not exist in the target runtime; the bridge falls back to a polite "endpoint pending" notice without raising an error.

node plugins/codex/scripts/loomi-bridge.mjs pet happy
node plugins/codex/scripts/loomi-bridge.mjs pet working

Failure modes (all return structured JSON, never throw):

Help the user customize their pet's appearance

The Codex bridge only drives the state — sprite overrides and theme folders are file-based and live outside the plugin. When the user asks to "change the pet's look" or "add a custom character", walk them through the file system. Do not try to write pet-config.json from the bridge; the bridge has no such command and the runtime's file watcher does the work.

Decision tree

1. "I just want the other built-in" → right-click Loomi → Theme → Fox / Theme → Capybara. Persisted in ~/.openloomi/pet-config.json under activeTheme. Long-press (~600 ms) is the fallback if contextmenu is swallowed by the host.
2. "I want my own character" → drop a folder at ~/.openloomi/pet-custom/<name>/ with PNGs named after the states. The watcher auto-discovers it within ~250 ms and the theme appears in the menu.
3. "I want to change just one sprite" → edit ~/.openloomi/pet-config.json's overrides map. Wins over both built-ins and custom-theme sprites for the matching (theme, state) pair.
4. "I want to make my theme the default" → set activeTheme to the custom theme's folder name.

What the bridge can and can't do

Filename conventions to communicate

When guiding the user through a custom theme, surface these conventions up-front:

• PNG only. .gif, .webp, .apng, .lottie are silently ignored.
• Bare or prefixed names both work. idle.png, loomi-idle.png, capybara-thinking.png, my-pack-sweeping.png all normalize correctly (case-insensitive).
• One recognizable state PNG is enough. The folder is registered as a theme as soon as it has ≥1 normalized state stem; missing states fall through to the active theme's idle sprite.
• Hidden / dot-prefixed folders are ignored. .git, .DS_Store etc.

Override JSON shape to communicate

The overrides map is camelCase on the wire — activeTheme, customThemesDir. Snake_case keys silently no-op the assignment (the unit test at apps/web/src-tauri/src/pet/theme.rs:499 pins the contract). Use the exact shape:

{
  "version": 1,
  "activeTheme": "fox",
  "customThemesDir": "~/.openloomi/pet-custom",
  "overrides": {
    "fox": {
      "idle": "/absolute/path/to/my-fox-idle.png"
    }
  }
}

Absolute paths only — the runtime routes them through tauri::convertFileSrc, so relative paths and ~/ prefixes will not resolve.

Common pitfalls to surface

• The folder doesn't show up — usually a missing or mis-named state PNG. Run them through the filename convention table.
• Override doesn't apply — host log line [loomi-pet/theme] failed to parse ~/.openloomi/pet-config.json means the JSON itself is malformed and defaults loaded. Host log line [loomi-pet/theme] failed to read <path>: <io error> means the path doesn't resolve.
• Theme menu tick is wrong / stuck — almost always a camelCase vs snake_case wire-format mismatch. The widget reads activeTheme, not active_theme.
• sleeping / sweeping rejected by /api/pet/state — those are watcher-only vocabulary; the runtime returns 400 invalid_state. The Codex bridge surfaces this as PET_FAILED (not INVALID_STATE) — distinguish it from a typo before reporting.

Codex-specific deltas vs the Claude plugin

What NOT to do from Codex

• Do not try to call POST /api/pet/state with sleeping or sweeping. The API rejects them; the watcher owns them.
• Do not try to write pet-config.json from the bridge. The bridge has no command for it; the file watcher owns updates.
• Do not invent new state names. CAPYBARA_STATES rejects them before any HTTP call.
• Do not claim support for non-PNG sprite formats. The asset pipeline is static PNG only.
• Do not say "I'll set up your custom theme for you" if you can only walk them through the file system. The watcher does the work; you guide.

See also

• Customize your Loomi Pet (user docs) — full guide to themes, custom folders, overrides, troubleshooting
• Pet API — POST /api/pet/state for external tools
• Attention Agent — the desktop pet as a whole
• The runtime source: apps/web/src-tauri/src/pet/theme.rs (custom themes + overrides), apps/web/src-tauri/src/pet/watcher.rs::map_state_to_pet (state resolution)
• Claude-side counterpart: plugins/claude/skills/openloomi-pet/SKILL.md

OpenLoomi

Use this skill when the user wants Codex to work with OpenLoomi as a local personal assistant, memory layer, or setup guide.

This skill is intentionally thin. It calls the local bridge and lets OpenLoomi own runtime execution, memory, connectors, settings, and secret storage.

Before taking action, check plugin readiness:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" setup-status

If the bridge returns ready: false, follow the reported nextAction. Do not ask the user to paste API keys, OAuth tokens, connector secrets, or OpenLoomi auth tokens into Codex chat.

OpenLoomi guest sessions are supported. A missing token is not a request for account registration or manual token entry. When the bridge reports initialize_openloomi_session or open_openloomi, initialize a guest/session through OpenLoomi-owned surfaces:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" initialize-session

For installation guidance, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" install-instructions

If the user asks to install OpenLoomi or explicitly approves installation, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" install-openloomi --confirm

The bridge resolves the official GitHub release artifact for the current platform and architecture automatically, downloads it, and installs it with the default installer path when automatic installation is supported. Only pass --artifact-url when the user explicitly provides an official allowlisted artifact URL as an override. Add --download-only only when the user asks to download without installing. Add --launch only when the user asks to use the interactive installer UI instead of default automatic installation. Add --sha256 "<official checksum>" only when the user wants to require a specific checksum; otherwise the bridge verifies GitHub release digest metadata when available.

For AI provider setup guidance, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" configure-ai-provider

You may pass non-secret preferences such as --provider, --base-url, and --model when the user provides them. Never pass --api-key, tokens, or other secrets. Secret entry must happen in an OpenLoomi-owned UI or interactive CLI surface.

AI provider readiness may come from environment variables or from OpenLoomi-owned UI/runtime settings. If the bridge reports AI_PROVIDER_STATUS_UNAVAILABLE, guide the user to open OpenLoomi so the local API can confirm whether provider settings exist. Do not ask the user to repeat API keys in Codex chat.

For bridge metadata, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" version

For available OpenLoomi workflows, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance

For workflow-specific guidance, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-loop
node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-memory
node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-connectors
node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" workflow-guidance --workflow openloomi-handoff

Use the thin wrapper skills when the user specifically asks for loop, memory, connector readiness, or handoff workflows. The plugin must not copy OpenLoomi connector, memory, loop, scheduling, or handoff persistence logic into Codex.

Launching the desktop app with the Codex runtime

When OpenLoomi is used from Codex, prefer the desktop Codex runtime so OpenLoomi can reuse the user's existing Codex CLI runtime instead of requiring a separate OpenLoomi AI provider setup for the first workflow.

When the user asks to make OpenLoomi spawn Codex as the native-agent executor, or diagnostics show that the desktop runtime is not using Codex, call:

node "$SKILL_DIR/../../scripts/loomi-bridge.mjs" codex-runtime-info

Show the returned platform-specific guidance, then ask the user to restart OpenLoomi and verify /api/native/providers reports defaultAgent: "codex".

Browser Automation with agent-browser

Installation

Global Installation (recommended)

Installs the native Rust binary for maximum performance:

npm install -g agent-browser
agent-browser install  # Download Chromium

This is the fastest option -- commands run through the native Rust CLI directly with sub-millisecond parsing overhead.

Quick Start (no install)

Run directly with npx if you want to try it without installing globally:

npx agent-browser install   # Download Chromium (first time only)
npx agent-browser open example.com

Note: npx routes through Node.js before reaching the Rust CLI, so it is noticeably slower than a global install. For regular use, install globally.

Homebrew (macOS)

brew install agent-browser
agent-browser install  # Download Chromium

Linux Dependencies

On Linux, install system dependencies:

agent-browser install --with-deps
# or manually: npx playwright install-deps chromium

Core Workflow

Every browser automation follows this pattern:

1. Navigate: agent-browser open <url>
2. Snapshot: agent-browser snapshot -i (get element refs like @e1, @e2)
3. Interact: Use refs to click, fill, select
4. Re-snapshot: After navigation or DOM changes, get fresh refs

agent-browser open https://example.com/form
agent-browser snapshot -i
# Output: @e1 [input type="email"], @e2 [input type="password"], @e3 [button] "Submit"

agent-browser fill @e1 "user@example.com"
agent-browser fill @e2 "password123"
agent-browser click @e3
agent-browser wait --load networkidle
agent-browser snapshot -i  # Check result

Command Chaining

Commands can be chained with && in a single shell invocation. The browser persists between commands via a background daemon, so chaining is safe and more efficient than separate calls.

# Chain open + wait + snapshot in one call
agent-browser open https://example.com && agent-browser wait --load networkidle && agent-browser snapshot -i

# Chain multiple interactions
agent-browser fill @e1 "user@example.com" && agent-browser fill @e2 "password123" && agent-browser click @e3

# Navigate and capture
agent-browser open https://example.com && agent-browser wait --load networkidle && agent-browser screenshot page.png

When to chain: Use && when you don't need to read the output of an intermediate command before proceeding (e.g., open + wait + screenshot). Run commands separately when you need to parse the output first (e.g., snapshot to discover refs, then interact using those refs).

Essential Commands

# Navigation
agent-browser open <url>              # Navigate (aliases: goto, navigate)
agent-browser close                   # Close browser

# Snapshot
agent-browser snapshot -i             # Interactive elements with refs (recommended)
agent-browser snapshot -i -C          # Include cursor-interactive elements (divs with onclick, cursor:pointer)
agent-browser snapshot -s "#selector" # Scope to CSS selector

# Interaction (use @refs from snapshot)
agent-browser click @e1               # Click element
agent-browser click @e1 --new-tab     # Click and open in new tab
agent-browser fill @e2 "text"         # Clear and type text
agent-browser type @e2 "text"         # Type without clearing
agent-browser select @e1 "option"     # Select dropdown option
agent-browser check @e1               # Check checkbox
agent-browser press Enter             # Press key
agent-browser scroll down 500         # Scroll page

# Get information
agent-browser get text @e1            # Get element text
agent-browser get url                 # Get current URL
agent-browser get title               # Get page title

# Wait
agent-browser wait @e1                # Wait for element
agent-browser wait --load networkidle # Wait for network idle
agent-browser wait --url "**/page"    # Wait for URL pattern
agent-browser wait 2000               # Wait milliseconds

# Capture
agent-browser screenshot              # Screenshot to temp dir
agent-browser screenshot --full       # Full page screenshot
agent-browser screenshot --annotate   # Annotated screenshot with numbered element labels
agent-browser pdf output.pdf          # Save as PDF

# Diff (compare page states)
agent-browser diff snapshot                          # Compare current vs last snapshot
agent-browser diff snapshot --baseline before.txt    # Compare current vs saved file
agent-browser diff screenshot --baseline before.png  # Visual pixel diff
agent-browser diff url <url1> <url2>                 # Compare two pages
agent-browser diff url <url1> <url2> --wait-until networkidle  # Custom wait strategy
agent-browser diff url <url1> <url2> --selector "#main"  # Scope to element

Common Patterns

Form Submission

agent-browser open https://example.com/signup
agent-browser snapshot -i
agent-browser fill @e1 "Jane Doe"
agent-browser fill @e2 "jane@example.com"
agent-browser select @e3 "California"
agent-browser check @e4
agent-browser click @e5
agent-browser wait --load networkidle

Authentication with State Persistence

# Login once and save state
agent-browser open https://app.example.com/login
agent-browser snapshot -i
agent-browser fill @e1 "$USERNAME"
agent-browser fill @e2 "$PASSWORD"
agent-browser click @e3
agent-browser wait --url "**/dashboard"
agent-browser state save auth.json

# Reuse in future sessions
agent-browser state load auth.json
agent-browser open https://app.example.com/dashboard

Session Persistence

# Auto-save/restore cookies and localStorage across browser restarts
agent-browser --session-name myapp open https://app.example.com/login
# ... login flow ...
agent-browser close  # State auto-saved to ~/.agent-browser/sessions/

# Next time, state is auto-loaded
agent-browser --session-name myapp open https://app.example.com/dashboard

# Encrypt state at rest
export AGENT_BROWSER_ENCRYPTION_KEY=$(openssl rand -hex 32)
agent-browser --session-name secure open https://app.example.com

# Manage saved states
agent-browser state list
agent-browser state show myapp-default.json
agent-browser state clear myapp
agent-browser state clean --older-than 7

Data Extraction

agent-browser open https://example.com/products
agent-browser snapshot -i
agent-browser get text @e5           # Get specific element text
agent-browser get text body > page.txt  # Get all page text

# JSON output for parsing
agent-browser snapshot -i --json
agent-browser get text @e1 --json

Parallel Sessions

agent-browser --session site1 open https://site-a.com
agent-browser --session site2 open https://site-b.com

agent-browser --session site1 snapshot -i
agent-browser --session site2 snapshot -i

agent-browser session list

Connect to Existing Chrome

# Auto-discover running Chrome with remote debugging enabled
agent-browser --auto-connect open https://example.com
agent-browser --auto-connect snapshot

# Or with explicit CDP port
agent-browser --cdp 9222 snapshot

Color Scheme (Dark Mode)

# Persistent dark mode via flag (applies to all pages and new tabs)
agent-browser --color-scheme dark open https://example.com

# Or via environment variable
AGENT_BROWSER_COLOR_SCHEME=dark agent-browser open https://example.com

# Or set during session (persists for subsequent commands)
agent-browser set media dark

Visual Browser (Debugging)

agent-browser --headed open https://example.com
agent-browser highlight @e1          # Highlight element
agent-browser record start demo.webm # Record session
agent-browser profiler start         # Start Chrome DevTools profiling
agent-browser profiler stop trace.json # Stop and save profile (path optional)

Local Files (PDFs, HTML)

# Open local files with file:// URLs
agent-browser --allow-file-access open file:///path/to/document.pdf
agent-browser --allow-file-access open file:///path/to/page.html
agent-browser screenshot output.png

iOS Simulator (Mobile Safari)

# List available iOS simulators
agent-browser device list

# Launch Safari on a specific device
agent-browser -p ios --device "iPhone 16 Pro" open https://example.com

# Same workflow as desktop - snapshot, interact, re-snapshot
agent-browser -p ios snapshot -i
agent-browser -p ios tap @e1          # Tap (alias for click)
agent-browser -p ios fill @e2 "text"
agent-browser -p ios swipe up         # Mobile-specific gesture

# Take screenshot
agent-browser -p ios screenshot mobile.png

# Close session (shuts down simulator)
agent-browser -p ios close

Requirements: macOS with Xcode, Appium (npm install -g appium && appium driver install xcuitest)

Real devices: Works with physical iOS devices if pre-configured. Use --device "<UDID>" where UDID is from xcrun xctrace list devices.

Diffing (Verifying Changes)

Use diff snapshot after performing an action to verify it had the intended effect. This compares the current accessibility tree against the last snapshot taken in the session.

# Typical workflow: snapshot -> action -> diff
agent-browser snapshot -i          # Take baseline snapshot
agent-browser click @e2            # Perform action
agent-browser diff snapshot        # See what changed (auto-compares to last snapshot)

For visual regression testing or monitoring:

# Save a baseline screenshot, then compare later
agent-browser screenshot baseline.png
# ... time passes or changes are made ...
agent-browser diff screenshot --baseline baseline.png

# Compare staging vs production
agent-browser diff url https://staging.example.com https://prod.example.com --screenshot

diff snapshot output uses + for additions and - for removals, similar to git diff. diff screenshot produces a diff image with changed pixels highlighted in red, plus a mismatch percentage.

Timeouts and Slow Pages

The default Playwright timeout is 60 seconds for local browsers. For slow websites or large pages, use explicit waits instead of relying on the default timeout:

# Wait for network activity to settle (best for slow pages)
agent-browser wait --load networkidle

# Wait for a specific element to appear
agent-browser wait "#content"
agent-browser wait @e1

# Wait for a specific URL pattern (useful after redirects)
agent-browser wait --url "**/dashboard"

# Wait for a JavaScript condition
agent-browser wait --fn "document.readyState === 'complete'"

# Wait a fixed duration (milliseconds) as a last resort
agent-browser wait 5000

When dealing with consistently slow websites, use wait --load networkidle after open to ensure the page is fully loaded before taking a snapshot. If a specific element is slow to render, wait for it directly with wait <selector> or wait @ref.

Session Management and Cleanup

When running multiple agents or automations concurrently, always use named sessions to avoid conflicts:

# Each agent gets its own isolated session
agent-browser --session agent1 open site-a.com
agent-browser --session agent2 open site-b.com

# Check active sessions
agent-browser session list

Always close your browser session when done to avoid leaked processes:

agent-browser close                    # Close default session
agent-browser --session agent1 close   # Close specific session

If a previous session was not closed properly, the daemon may still be running. Use agent-browser close to clean it up before starting new work.

Ref Lifecycle (Important)

Refs (@e1, @e2, etc.) are invalidated when the page changes. Always re-snapshot after:

• Clicking links or buttons that navigate
• Form submissions
• Dynamic content loading (dropdowns, modals)

agent-browser click @e5              # Navigates to new page
agent-browser snapshot -i            # MUST re-snapshot
agent-browser click @e1              # Use new refs

Annotated Screenshots (Vision Mode)

Use --annotate to take a screenshot with numbered labels overlaid on interactive elements. Each label [N] maps to ref @eN. This also caches refs, so you can interact with elements immediately without a separate snapshot.

agent-browser screenshot --annotate
# Output includes the image path and a legend:
#   [1] @e1 button "Submit"
#   [2] @e2 link "Home"
#   [3] @e3 textbox "Email"
agent-browser click @e2              # Click using ref from annotated screenshot

Use annotated screenshots when:

• The page has unlabeled icon buttons or visual-only elements
• You need to verify visual layout or styling
• Canvas or chart elements are present (invisible to text snapshots)
• You need spatial reasoning about element positions

Semantic Locators (Alternative to Refs)

When refs are unavailable or unreliable, use semantic locators:

agent-browser find text "Sign In" click
agent-browser find label "Email" fill "user@test.com"
agent-browser find role button click --name "Submit"
agent-browser find placeholder "Search" type "query"
agent-browser find testid "submit-btn" click

JavaScript Evaluation (eval)

Use eval to run JavaScript in the browser context. Shell quoting can corrupt complex expressions -- use --stdin or -b to avoid issues.

# Simple expressions work with regular quoting
agent-browser eval 'document.title'
agent-browser eval 'document.querySelectorAll("img").length'

# Complex JS: use --stdin with heredoc (RECOMMENDED)
agent-browser eval --stdin <<'EVALEOF'
JSON.stringify(
  Array.from(document.querySelectorAll("img"))
    .filter(i => !i.alt)
    .map(i => ({ src: i.src.split("/").pop(), width: i.width }))
)
EVALEOF

# Alternative: base64 encoding (avoids all shell escaping issues)
agent-browser eval -b "$(echo -n 'Array.from(document.querySelectorAll("a")).map(a => a.href)' | base64)"

Why this matters: When the shell processes your command, inner double quotes, ! characters (history expansion), backticks, and $() can all corrupt the JavaScript before it reaches agent-browser. The --stdin and -b flags bypass shell interpretation entirely.

Rules of thumb:

• Single-line, no nested quotes -> regular eval 'expression' with single quotes is fine
• Nested quotes, arrow functions, template literals, or multiline -> use eval --stdin <<'EVALEOF'
• Programmatic/generated scripts -> use eval -b with base64

Configuration File

Create agent-browser.json in the project root for persistent settings:

{
  "headed": true,
  "proxy": "http://localhost:8080",
  "profile": "./browser-data"
}

Priority (lowest to highest): ~/.agent-browser/config.json < ./agent-browser.json < env vars < CLI flags. Use --config <path> or AGENT_BROWSER_CONFIG env var for a custom config file (exits with error if missing/invalid). All CLI options map to camelCase keys (e.g., --executable-path -> "executablePath"). Boolean flags accept true/false values (e.g., --headed false overrides config). Extensions from user and project configs are merged, not replaced.

Deep-Dive Documentation

Ready-to-Use Templates

./templates/form-automation.sh https://example.com/form
./templates/authenticated-session.sh https://app.example.com/login
./templates/capture-workflow.sh https://example.com ./output

When to Apply

• User wants to access or interact with external apps (Gmail, Slack, GitHub, Notion, etc.)
• User wants to automate a task using an external service (send email, create issue, post message)
• Building an AI agent or app that integrates with external tools
• Multi-user apps that need per-user connections to external services

Setup

Check if the CLI is installed; if not, install it:

curl -fsSL https://composio.dev/install | bash

After installation, restart your terminal or source your shell config, then authenticate:

composio login       # OAuth; interactive org/project picker (use -y to skip)
composio whoami      # verify org_id, project_id, user_id

For agents without direct browser access: composio login --no-wait | jq to get URL/key, share URL with user, then composio login --key <cli_key> --no-wait once they complete login.

1. Use Apps via Composio CLI

Use this when: The user wants to take action on an external app directly — no code writing needed. The agent uses the CLI to search, connect, and execute tools on behalf of the user.

Key commands (new top-level aliases):

• composio search "<query>" — find tools by use case
• composio execute "<TOOL_SLUG>" -d '{...<input params>}' — execute a tool
• composio link [toolkit] — connect a user account to an app (agents: always use --no-wait for non-interactive mode)
• composio listen — listen for real-time trigger events

Typical workflow: search → link (if needed) → execute

Full reference: Composio CLI Guide

2. Building Apps and Agents with Composio

Use this when: Writing code — an AI agent, app, or backend service that integrates with external tools via the Composio SDK.

Run this first inside the project directory to set up the API key:

composio init

Full reference: Building with Composio

cua-driver

Orchestrates macOS app automation via cua-driver. Whenever a user asks to drive a native macOS app, follow the loop in this skill rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it.

The no-foreground contract — read this first

The user's frontmost app MUST NOT change. This is the whole reason cua-driver exists. Users pay for the right to keep typing in their editor while an agent drives another app in the background. Violate this rule and every other nice property the driver gives you (no cursor warp, no Space switch, no window raise) stops mattering — you just shipped the Accessibility Inspector with extra steps.

Before running any shell command, ask: "does this raise, activate, foreground, or make-key any app?" If yes, don't run it. Every one of the commands below activates the target on macOS and is therefore forbidden unless the user explicitly asked for frontmost state:

• Every form of the open CLI — open -a <App>, open -b <bundle-id>, open <file>, open <path-to-App.app>, open <url> — always activates. macOS routes all forms through LaunchServices, which unhides and foregrounds the target regardless of whether you passed an app name, a bundle id, a document, a URL, or the bundle path itself. The activation happens even when the only intent was "start the process." Never use open for any app launch. This includes launching a just-built .app from a local build dir (e.g. open build/Build/Products/Debug/MyApp.app) — resolve the CFBundleIdentifier from Info.plist and use launch_app with that id. See "The narrow carve-out" below for why launch_app is safe even when the app internally calls NSApp.activate.

• osascript -e 'tell application "X" to activate' — activates by design. Same for ... to open <file>, ... to launch, and anything with activate in the tell block.

• osascript -e 'tell application "System Events" to ... frontmost' in a mutating form (setting frontmost rather than reading it).

• AppleScript files that invoke activate, launch, or open against the target app.

• cliclick (moves the user's real cursor to the target coords before clicking — a focus-steal-equivalent even if the app's window state is unchanged).

• CGEventPost with cghidEventTap targeting a coordinate over a different app's window (warps the cursor, possibly activates on hit).

• AppleScriptTask, NSAppleScript, Process wrapping osascript that contains any of the above.

• NSRunningApplication.activate(options:) called from your own helper binary — same class.

• Dock clicks and any open invocation (see the first bullet — every form of open goes through LaunchServices which activates, full stop).

• Keyboard shortcuts that semantically mean "focus here" — most notably Chrome / Safari / Arc's ⌘L (focus omnibox) and Finder's ⌘⇧G (Go to Folder). These aren't pure key events — the receiving app interprets "user wants to type here" as activation intent and raises its window to be key. Even when delivered to a backgrounded pid via hotkey, the downstream app pulls focus. For omnibox navigation specifically, the correct path is launch_app({bundle_id: "com.google.Chrome", urls: ["https://…"]}) — no omnibox dance, no ⌘L, no focus-steal. Do NOT try set_value on the omnibox: Chrome's commit logic requires a "user-typed" signal that neither an AX value write nor CGEvent.postToPid keystrokes supply from a backgrounded pid — the URL lands in the field but Return fires as a no-op. See WEB_APPS.md → "Navigate to a URL" for the full pattern. The general principle: a shortcut that says "put my cursor inside this app" is a focus-steal; a shortcut that says "do this thing" (copy, save, quit) is fine.

• Tab-switching shortcuts in browsers (⌘1..⌘9, ⌘], ⌘[, ⌘⇧[, ⌘⇧]) are visibly disruptive even when delivered to a backgrounded pid. The app's key handler processes the shortcut, the window re-renders the new tab's content, the user sees their tabs flipping. There is no AX-only workaround: page content (HTML, form state, AXWebArea) populates only for the focused tab; inspecting a background tab requires activating it, which is the visible flip. Observed with Dia; the same mechanic applies to every Chromium-family browser (Chrome, Arc, Brave, Edge).

  Prefer the windows-over-tabs pattern: for each URL you need to drive backgrounded, use launch_app({bundle_id, urls: [url]}) — browsers open each URL in a new window. Each window has its own window_id, its own AX tree, and can be inspected / interacted with via element_index without activating or switching anything. Tabs are a UX grouping for humans; cua-driver workflows should default to windows. See WEB_APPS.md → "Tabs vs windows" for the full pattern.

  Tab-title enumeration (read-only) IS safe — walk a window's toolbar AX tree for AXTab / AXRadioButton children and read their AXTitles. Tab switching (activating one) is not.

Reading frontmost state is fine (osascript -e 'tell application "System Events" to get name of first application process whose frontmost is true'). Mutating it is not.

Corollary — the AXMenuBar rule. AXMenuBarItem + AXPick dispatches at the AX layer regardless of which app is frontmost, but macOS's on-screen menu bar always belongs to the frontmost app. If you drive a backgrounded app's menu bar, the AX call succeeds but the viewer sees the dispatch rendered over the frontmost app's menu bar — confusing in any observed session and routinely a silent no-op too, because action menu items go DISABLED when their owning app isn't the key window. So: only use menu-bar navigation when the target is already frontmost. For backgrounded targets, read state via in-window AX (window title, toolbar AXStaticText) and dispatch via in-window element_index or pixel clicks — both paths are frontmost-insensitive. Full rationale in "Navigating native menu bars" below.

"Open <app>" in user speech means launch, not activate. cua-driver launch_app is the one correct path for process startup — it's idempotent (no-op on a running app), returns the pid, and has an internal FocusRestoreGuard that catches NSApp.activate(ignoringOtherApps:) calls the target makes during application(_:open:) and clobbers the frontmost back to what it was before the launch. That guard is why launch_app with urls (e.g. {"bundle_id": "com.colliderli.iina", "urls": ["~/video.mp4"]}) is safe even for apps that normally foreground on media-load (Chrome, Electron, media players).

Defaults — always prefer cua-driver over shell shims

Default transport is the cua-driver CLI — Bash shelling out to cua-driver <tool-name> '<JSON-args>'. MCP tools (prefix mcp__cua-driver__*) only when the user explicitly asks for them. CLI wins because it picks up rebuilds instantly, failures are easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to click(...), get_window_state(...) etc. in this skill means cua-driver click '{...}' — translate to MCP form only when MCP is requested.

Claude Code computer-use compatibility mode

For normal Claude Code use, keep the default CLI or cua-driver MCP server path above. If the user explicitly wants Claude Code's vision/computer-use-style flow, they can register:

claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat

Observation: Claude Code vision flows appear to treat a screenshot MCP tool as the image-grounding anchor. This compatibility mode keeps the normal CuaDriver tools and changes only screenshot. The compatibility screenshot requires pid and window_id, captures only that target window, and returns the window-local pixel coordinate frame. Start with launch_app or list_windows, then call screenshot({pid, window_id}); do not assume desktop coordinates or a full-screen capture.

Use MCP for this Claude Code vision/computer-use-style path. Do not shell out to cua-driver screenshot as a substitute: CLI screenshots still work as CuaDriver calls, but they do not expose the mcp__cua-computer-use__screenshot tool name that Claude Code appears to use as the image-grounding cue.

Intent → tool mapping. If you find yourself reaching for the right column, something has gone wrong — re-read "The no-foreground contract" above:

The narrow carve-out

The only legitimate use of osascript -e 'tell app X to activate' is when the user explicitly asked for frontmost state ("bring Chrome to the front", "make it frontmost", "I want to see X"). Reaching for it because a tool call returned something confusing is wrong — that's the skill's classic foot-in-the-door failure mode and it steals focus every time.

When a cua-driver call surprises you, diagnose cua-driver first:

• Tiny screenshot / empty tree_markdown? Check cua-driver get_config → capture_mode. Default "som" returns both the AX tree and screenshot. "vision" omits the AX tree (PNG only), "ax" omits the PNG. If a snapshot lacks a tree, capture_mode is almost certainly "vision" — either reason purely from the PNG or flip to "som" / "ax" via set_config.
• has_screenshot: false? The window capture failed (transient race against a close, or the window has no backing store yet). Re-snapshot; if persistent, pick a different window_id via list_windows.
• Invalid element_index / No cached AX state? You either skipped get_window_state this turn or passed a different window_id than the one the snapshot cached against. The cache is keyed on (pid, window_id) — indices don't carry across windows of the same app. Re-snapshot with the same window_id you're about to click in.
• Sparse Chromium AX tree? Retry get_window_state once — the tree populates on second call.

Only after those are ruled out, and only if the user's action genuinely needs frontmost state, fall through to the activate fallback. Always name the focus steal in your response ("I'll briefly bring Chrome to the front because …").

Self-check pattern

Before every Bash call whose command line touches any macOS app (launching, opening, clicking, typing, scripting, screenshotting), run the self-check:

1. Does this command foreground the target? If yes — stop and translate to the cua-driver equivalent from the mapping table.
2. Does this command move the user's real cursor? (cliclick, any CGEventPost at cghidEventTap over another app's window). If yes — stop; use click({pid, x, y}) which routes per-pid via SkyLight and never warps the cursor.
3. Does this command bypass cua-driver entirely? (osascript mutating GUI state, AppleScript files, external helpers.) If yes — stop; find the cua-driver tool that does the intent.

If all three are "no," the command is safe. If you can't answer, default to stop and ask rather than proceed. A single open -a run by accident kills the demo, the trust, and the user's in-flight editor state.

Prerequisites — check before starting

1. cua-driver is on $PATH (which cua-driver). If not, point the user at scripts/install-local.sh and stop.
2. Run cua-driver check_permissions (with the daemon up — see step 3). The default behavior also raises the system permission dialogs for any missing grants, so the user can grant on the spot. If either grant still reads false after that (user dismissed the dialog), tell them to open System Settings → Privacy & Security and grant Accessibility and Screen Recording to CuaDriver.app, then stop. Pass '{"prompt":false}' for a purely read-only status check that won't steal focus.
3. Start the daemon with open -n -g -a CuaDriver --args serve (the recommended form — goes through LaunchServices so TCC attributes the process to CuaDriver.app). cua-driver serve & also works; the CLI auto-relaunches through open -n -g -a CuaDriver when it detects a wrong-TCC context (any IDE-spawned shell: Claude Code, Cursor, VS Code, Conductor). Verify with cua-driver status.

Using cua-driver from the shell

Tool names are snake_case, management subcommands are kebab-case — no ambiguity. Tools invoked as cua-driver <tool-name> '<JSON-args>'. Management subcommands:

• open -n -g -a CuaDriver --args serve — start persistent daemon (required for element_index workflows; without it each CLI invocation spawns a fresh process and the per-pid element cache dies between calls). cua-driver serve & also works — the CLI auto-relaunches via open when the shell's TCC context is wrong. Pass --no-relaunch / CUA_DRIVER_NO_RELAUNCH=1 to opt out.
• cua-driver stop / status
• cua-driver list-tools, describe <tool>
• cua-driver recording start|stop|status — see RECORDING.md

Canonical multi-step workflow:

open -n -g -a CuaDriver --args serve
cua-driver launch_app '{"bundle_id":"com.apple.calculator"}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
cua-driver get_window_state '{"pid":844,"window_id":10725}'
cua-driver click '{"pid":844,"window_id":10725,"element_index":14}'
cua-driver stop

Agent cursor overlay

Visual cursor overlay for demos and screen recordings. Default: enabled. Toggle with cua-driver set_agent_cursor_enabled '{"enabled":true|false}'. A triangle pointer Bezier-glides to each click target, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs: set_agent_cursor_motion takes any subset of start_handle, end_handle, arc_size, arc_flow, spring — tuneable at runtime, persisted to config.

Requires an AppKit runloop, which cua-driver serve / mcp bootstraps. One-shot CLI invocations skip the overlay entirely.

The core invariant — snapshot before AND after every action

Every action MUST be bracketed by get_window_state(pid, window_id):

• Before — the pre-action snapshot resolves the element_index you're about to use. Indices from previous turns are stale; the server replaces the element index map on every snapshot, keyed on (pid, window_id). Indices from turn N don't resolve in turn N+1, and indices from window A don't resolve against window B of the same app. Skip this and element-indexed actions fail with No cached AX state.
• After — the post-action snapshot verifies the action actually landed. Without it you can't tell a silent no-op from a real effect. The AX tree change (new value, new window, disappeared menu, disabled button, etc.) is your evidence that the action fired. If nothing changed, the action probably failed silently — say so, don't assume success.

This applies to pixel clicks too — re-snapshot after to confirm the click landed on the intended target.

Why window selection is the caller's job now

get_app_state used to pick a window for you via a max-area heuristic that returned the wrong surface on apps with large off-screen utility panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432 off-screen) out-area'd the visible 320×240 player window, so get_app_state(pid) screenshot'd the invisible panel and clicks landed there silently. The new get_window_state(pid, window_id) makes the caller name the window explicitly — the driver validates that the window belongs to the pid and is on the current Space, then snapshots exactly what was asked for. Enumerate candidates via list_windows or read the windows array launch_app already returns.

Behavior matrix

Two orthogonal axes shape what the agent can do.

capture_mode → addressing mode

vision was renamed from screenshot — the old name still decodes as a deprecated alias, so an on-disk "capture_mode": "screenshot" keeps working. Default is som so element_index clicks work the first time a user calls get_window_state; the other modes are opt-in when the caller specifically doesn't want one half of the work. Note the tool named screenshot is separate (raw PNG, no AX walk) and unrelated to the capture mode.

When a snapshot looks wrong (tiny screenshot / empty tree), check cua-driver get_config for capture_mode before anything else.

Pure-vision mode has its own caveats — Claude Code's vision pipeline downsamples dense text aggressively, so pixel grounding takes multiple correction cycles on text-heavy UIs. Read SCREENSHOT.md before driving anything in that mode; it documents the iterate/annotate/verify recipe plus the JPEG-over-PNG finding.

Window state → what works

Critical cell — minimized + keyboard commit. The keystroke reaches the app but AX focus doesn't propagate to renderer focus on a minimized window. Workarounds in order of preference: set_value to write the field's entire value directly, or AX-click a commit-equivalent button (Go, Submit, checkbox). Tell the user the window needs to un-minimize only as a last resort.

The canonical loop

launch_app(target)
  → pick window_id from the returned `windows` array
    (or call list_windows(pid) separately)
  → get_window_state(pid, window_id)
    → [act]  # every action also takes (pid, window_id)
  → get_window_state(pid, window_id) → verify

launch_app now returns a windows array alongside the pid, so the common case collapses to two calls (launch_app → get_window_state) without a separate list_windows hop.

1. Resolve target pid — always via launch_app

Always start with launch_app, whether or not the target is already running. It's idempotent (relaunching returns the existing pid with no side effects) and gives you the pid in one call — no list_apps hop.

• launch_app({bundle_id: "com.apple.finder"}) — preferred, unambiguous.
• launch_app({name: "Calculator"}) — when bundle_id isn't known.

launch_app is a hidden-launch primitive by design — that's the entire point of cua-driver: agents drive apps in the background while the user keeps typing in their real foreground app. The target's window is initialized (AX tree fully populated, clickable via element_index, the pid appears in list_apps) but not drawn on screen. The driver never activates or unhides apps on its own; that would violate the no-foreground contract the whole driver exists to protect.

If the user explicitly wants the window visible (usually for a demo or recording), they unhide it themselves — Dock click, Cmd-Tab, or Spotlight. Do not reach for open / osascript activate as a shortcut to make the window visible; those paths break the backgrounded invariant on every call, not just the call that "needed" the foreground. Say out loud what the user needs to do ("click the Todo app in your Dock to bring it forward") and let them do it.

Never shell out to any form of open (including open <path-to-App.app> for a just-built binary — resolve the bundle id from Info.plist and use launch_app with that), osascript 'tell app … to launch/open', or similar. Those paths activate the target, bypass the driver's focus-restore guard, and require a Bash permission prompt the agent loop shouldn't be burning on app launch. See "Prefer cua-driver tools over shell shims" above for the full intent → tool mapping.

list_apps is for app-level discovery (answering "what's installed / running / frontmost?") — not part of the core action loop. Skip it in the loop. For window-level questions — "does this app have a visible window?", "which Space is this window on?", "which of this pid's windows is the main one?" — call list_windows instead; the app record doesn't carry window state on purpose. In the common single-window case you can skip list_windows entirely and read the windows array that launch_app already returned.

2. Snapshot and act by element_index

Call get_window_state({pid, window_id}) with the window_id from launch_app's windows array (or a fresh list_windows({pid}) if you're interacting with a long-lived process). The default som capture_mode returns both the AX tree and screenshot, so the canonical loop works immediately without any config change. The rest of this section walks through som mode. If you're in vision mode (PNG only, no AX tree), flip back: cua-driver set_config '{"key": "capture_mode", "value": "som"}'.

In som mode (the default) the response carries:

• tree_markdown — every actionable element tagged [N]. That N is the element_index. The tree can be very large (Finder is ~1600 elements, ~190 KB); when it exceeds token limits the MCP harness saves it to a file and returns the path. Use Bash + jq -r '.tree_markdown' + grep to pull the section you need.
• screenshot_file_path — absolute path to the saved screenshot when screenshot_out_file was passed. Absent otherwise.
• screenshot_width / _height / _scale_factor — dimensions of the captured image. Present whenever a screenshot was taken. Getting the screenshot as a file (CLI and context-constrained agents):

# write to file — stdout stays readable (AX tree / summary only, no base64)
cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'

# CLI --screenshot-out-file flag is equivalent and works for all capture modes
cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg

Pass screenshot_out_file when using get_window_state via CLI or from an agent whose context window can't absorb ~31 KB of inline base64 (e.g. OpenCode with a local Ollama model). The MCP image content block is omitted from the response when this param is set — the model receives only the AX tree and screenshot_file_path, then reads the image from disk.

Reason over both the tree AND the screenshot — they're complementary, not redundant. In som mode every turn's get_window_state gives you both halves and you should pull signal from each:

• The AX tree tells you what's clickable — roles, labels, element_index handles, advertised actions, parent-child structure. This is the ground truth for dispatching.
• The screenshot tells you which one — the tree often has many buttons with similar or empty labels ("Delete", "OK", anonymous UUID-labeled buttons, five AXStaticText = " "), and visual context disambiguates. Captions, colors, layout relationships visible in pixels often don't show up in the AX tree at all (especially in Chromium / Electron / web content).

Canonical pattern: look at the screenshot to decide "the blue Subscribe button on the top-right of the video card", then walk the tree to find the matching AXButton and dispatch by its element_index. Don't try to do it from just the tree — you'll pick the wrong element when labels repeat. Don't try to do it from just the screenshot — you lose the reliable AX-action path and the safe backgrounded-dispatch.

Reach for pixel coordinates only when the target is a canvas / video / WebGL / custom-drawn surface that isn't in the AX tree (see Pixel-coordinate clicks below).

The actions=[...] list on each element is advisory, not authoritative. cua-driver does not pre-flight check against it — click({pid, element_index}) always attempts AXPress (or the action you pass) and surfaces whatever the target returns. Many apps accept AXPress on elements that don't advertise it — Chrome's omnibox suggestion AXMenuItem is a live example. Try the click first — pivot only on the returned AX error code.

Dispatch table (every row assumes a (pid, window_id) pair from the last get_window_state; window_id is required alongside element_index, ignored on pixel-only forms unless you want to anchor the conversion against a specific window):

All keyboard/text primitives require pid. There is no frontmost-routed variant — every key goes to the named target via CGEvent.postToPid, so the driver cannot leak keystrokes into the user's foreground app.

Why element_index is the primary path: works on hidden / occluded / off-Space windows, no focus steal, stable across rebuilds, labels tell you what you're clicking. Reach for pixel coordinates only when AX can't.

Pixel-coordinate clicks

The pixel path (click({pid, x, y})) is for surfaces the AX tree doesn't reach — canvases, video players, WebGL, custom-drawn controls. Coords are window-local screenshot pixels (same space as the PNG get_window_state returns). Top-left origin, y-down. The driver handles screen-point conversion internally. Passing window_id alongside x, y is optional but recommended — it pins the coordinate conversion to the window whose screenshot produced the pixel, rather than the driver's heuristic choice.

Reading coordinates from the PNG

PNGs returned by get_window_state are capped at 1568 px long-side by default (max_image_dimension config), matching Anthropic's multimodal-vision downsampling limit. That means the image the model reasons over and the image the click tool's coordinate system lives in are the same resolution — just look at the PNG, pick a pixel, click at that pixel. No scaling math.

This is the default because the mismatch between "rendered thumbnail" and "native PNG" was a recurring coord-estimation footgun. If you opt out (explicit max_image_dimension=0 for pixel-perfect verification flows), the old rule applies: don't eyeball coords from whatever your client renders — it may be 2-4× smaller than the PNG on disk, and a 2% error in thumbnail space becomes ~80 px in the real image. Use the crosshair recipe below against the full-resolution file in that case.

1. get_window_state({pid, window_id}) returns an image capped at 1568 long-side (default) plus its dimensions (screenshot_width / screenshot_height). Write the bytes to disk with --screenshot-out-file <path> in any capture mode — works identically in vision (where it's the only way) and som (where it sidesteps the jq + base64 dance on the spliced screenshot_png_b64 field).
2. You are a multimodal model — look at the PNG. Since the PNG matches what you see, pick the target pixel directly. No fractional math needed.
3. When precision matters (small targets, dense UIs), draw a crosshair on the image (do not crop — cropping loses the coordinate system and requires error-prone offset math) and show it before clicking:

from PIL import Image, ImageDraw
img = Image.open('/tmp/shot.png')
draw = ImageDraw.Draw(img)
x, y = <your_coordinate>
r = 18
draw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)
draw.line([x-30, y, x+30, y], fill='red', width=3)
draw.line([x, y-30, x, y+30], fill='red', width=3)
img.save('/tmp/shot_annotated.png')

4. Only dispatch the click after the user (or your own re-read of the annotated image) confirms the crosshair is on target.

Addressing variants

• click({pid, x, y}) — single left-click.
• click({pid, x, y, count: 2}) — double-click.
• click({pid, x, y, modifier: ["cmd"]}) — cmd-click. Accepts any subset of cmd/shift/option/ctrl.
• right_click({pid, x, y}) — also takes modifier.

The pixel path animates the agent cursor overlay but never warps the real cursor. If the pid has no on-screen window the call errors with pid X has no on-screen window — you need a visible window to anchor the conversion.

How the pixel click is dispatched

The recipe is the backgrounded "noraise" sequence: yabai's focus-without-raise SLPS event records followed by an off-screen user-activation primer and the real click, all stamped via SLEventPostToPid. The target app becomes AppKit-active for event routing but its window does not rise to the front of the z-stack, and macOS's "switch to Space with windows for app" follow is suppressed. Full mechanics in Sources/CuaDriverCore/Input/MouseInput.swift (clickViaAuthSignedPost) and the companion FocusWithoutRaise.swift.

Known limits

• Chromium <video> play/pause: pixel click is often rejected by HTML5's click-to-play handler on some builds. Use keyboard instead: press_key({pid, key: "k"}) (YouTube) or press_key({pid, key: "space"}) (generic). Keyboard events travel through a different auth envelope.
• Pixel right-click on Chromium web content coerces to a left-click — a known Chromium renderer-IPC limitation that affects every non-HID-tap synthesis path. For context menus on AX-addressable elements (links, buttons, toolbar items), use right_click({pid, element_index}) instead.

Canvases, viewports, games (Blender, Unity, GHOST, Qt, wxWidgets)

Apps whose main surface is an OpenGL / Metal / Qt / wxWidgets viewport expose no useful AX tree — the whole surface is one opaque AXGroup or AXWindow from AX's perspective. Per-pid event paths (SLEventPostToPid, CGEvent.postToPid) are filtered by the viewport's own event-source check and silently dropped — the event loop wants "real HID origin".

The working pattern:

1. Bring the target frontmost (a brief osascript activate is acceptable here — this is the carve-out the skill's osascript gate allows).
2. CGEvent.post(tap: .cghidEventTap) with a leading mouseMoved event (~30 ms before the click). cua-driver click when the target is frontmost automatically takes this path.
3. Accept that the real cursor visibly moves — cghidEventTap is the system HID stream, the cursor warps to the click point.

There is no backgrounded path that reaches these apps today.

Navigating native menu bars (AXMenuBar)

Only drive the menu bar when the target app is frontmost. This is the single most-misused cua-driver capability. If the target is backgrounded, don't reach for AXMenuBarItem + AXPick — use in-window element_index or pixel clicks instead. Two reasons, one functional and one perceptual:

• Functional: menu items that touch document/playback/editor state go DISABLED when their owning app isn't the key window (Preview rotate, IINA speed change, most editor commands). AXPick
  • AXPress will dispatch successfully from the driver's side but no-op at the target — you get a silent false-pass.
• Perceptual (matters for demos, screen recordings, and anything the user watches live): macOS's screen-rendered menu bar always belongs to the frontmost app. AXPick on a backgrounded app's AXMenuBarItem dispatches to that app's per-process menu at the AX layer, but any visible menu render happens over the frontmost app's menu bar — the viewer sees an IINA submenu flashing on top of Chrome's menus, which reads as "the agent clicked the wrong app." The AX call was correct; the frame the user sees is not. For recorded or observed sessions, this is an integrity bug even though it's not a correctness bug.

Good decision rule: if the target is not already frontmost, do not use AXMenuBarItem at all. For reading in-window state, snapshot the window AX tree — most apps expose the same state via an in-window AXStaticText, title bar, or toolbar. For dispatching actions, use in-window element_index (buttons, toolbar items) or pixel clicks on in-window controls — both dispatch via AppKit's window-under-pointer hit-test and are not frontmost-gated.

When the target IS frontmost, the menu-bar flow below is fine and the canonical path for menus.

The two-snapshot pattern (target frontmost only)

Menu contents are a two-snapshot flow. Closed AXMenu subtrees are deliberately skipped during snapshot — otherwise every app's File / Edit / View hierarchy plus every Recent Items macOS has ever seen would inflate the tree 10-100x. But once a menu is open, its AXMenuItem children do receive element_index values so you can click them normally.

1. Find the [N] AXMenuBarItem "<Menu Name>" in the tree.
2. click({pid, element_index: N, action: "pick"}) — menu bar items implement AXPick ("open my submenu"), not AXPress. Using the default action on an AXMenuBarItem is a no-op.
3. Re-snapshot. The expanded menu's items now appear under the bar item as [M] AXMenuItem "<Item Name>".
4. Click the target item — most items respond to AXPress (default action). Submenus nest under the item and are walked the same way.
5. Re-snapshot and verify.

If you ever need to back out without selecting, press_key({pid, key: "escape"}) closes the open menu. Leaving a menu expanded between turns poisons subsequent snapshots for that pid.

Commands gated on the target being frontmost

Some menu items and global shortcuts (Preview's Tools → Rotate Right, ⌘R; anything in the View menu that manipulates the current document; most editor commands) are disabled unless the target app is the key / frontmost window. You'll see it in the AX tree as DISABLED on the menu item even though the user's intent is obviously valid.

Before activating, confirm you're in this narrow case — the menu item still reads DISABLED after a fresh snapshot AND the action the user requested genuinely requires frontmost (Preview rotate, View menu document manipulation, editor commands). If either check fails, don't activate.

When both checks pass, the driver has no activate tool (deliberately — the whole point is backgroundable control), so this is the one legitimate osascript fallback:

osascript -e 'tell application "<App Name>" to activate'

Then re-snapshot — the menu item loses its DISABLED tag — and click({action: "pick"}) the item. Alternatively, a hotkey call delivered to the now-frontmost app works for the shortcut form (⌘R, ⌘+, etc.).

Always name the focus steal in your response so the user isn't surprised — "Briefly activating Preview to enable Tools → Rotate Right" or similar. Don't silently steal focus. You don't need to restore the previous frontmost afterwards unless the user asks — they can cmd-tab back.

Web-rendered apps (browsers, Electron, Tauri)

For Chrome / Edge / Brave / Arc / Safari, Electron apps (Slack, VSCode, Notion, Discord), and Tauri apps — see WEB_APPS.md.

Covers: sparse AX tree population (retry-once pattern for Chromium), URL navigation via omnibox suggestions, the set_value workaround for keyboard commits on minimized windows (Return silently no-ops — symptom is a macOS system beep; use set_value or click a clickable equivalent), scrolling via synthetic PageUp/Down keystrokes, in-page clicks, and typing into web inputs.

Chromium web content specifically also coerces right_click back to left — use element_index for AX-addressable targets and accept the limit otherwise.

Browser JS primitives — page tool and get_window_state(javascript=)

When the AX tree doesn't expose the data you need (common in Chromium/Electron — the tree is sparse for web content), use the page tool or the javascript param on get_window_state to query the DOM directly via Apple Events. Requires "Allow JavaScript from Apple Events" to be enabled — see WEB_APPS.md for the setup path.

Three actions on the page tool:

• page({pid, window_id, action: "get_text"}) — returns document.body.innerText. Fastest way to read page content, prices, article text, or any raw text the AX tree truncates or omits.

• page({pid, window_id, action: "query_dom", css_selector: "a[href]", attributes: ["href"]}) — runs querySelectorAll and returns each match's tag, text, and requested attributes as a JSON array. Use for table rows, link hrefs, data attributes, structured page data.

• page({pid, window_id, action: "execute_javascript", javascript: "..."}) — raw JS. Wrap in an IIFE with try-catch. Don't use this for elements already indexed by get_window_state — click and set_value are more reliable there.

Co-located read — get_window_state with javascript:

get_window_state({pid, window_id, javascript: "document.title"})

Runs the JS and appends the result as a ## JavaScript result section alongside the AX snapshot — one round-trip instead of two. Use this when you need both the element tree (for subsequent clicks) and some page data in the same turn.

Decision rule — AX vs JS:

Supported backends:

Electron sandbox note: SIGUSR1 connects to the Node.js main process. Sandboxed Electron apps (VS Code, Cursor) strip require and Electron APIs there. Useful for: process.env, process.versions, process.cwd(), process.pid. For full DOM/renderer access, launch the app with --remote-debugging-port=9222 — cua-driver will detect and prefer the page target automatically.

Arc returns no values; Firefox has no JS-via-AppleEvents support — see WEB_APPS.md for the full matrix.

3. Re-snapshot and verify — mandatory

Always call get_window_state({pid, window_id}) after the action. This isn't optional verification — it's the second half of the snapshot invariant.

Check the AX tree diff: a changed value, a new element, a new window, or the disappearance of the thing you just clicked (menus collapse after selection, buttons may become disabled, etc.). If nothing changed, the action likely failed silently — tell the user what you attempted and what you observed, don't paper over with "done" language. Agents that skip this step report success on silently-dropped actions — the single most common failure mode.

Recording trajectories

Session-scoped action recording + replay, for demos, regressions, and training data. Only invoke when the user explicitly asks to record a session — the skill does not auto-enable this. CLI surface: cua-driver recording start|stop|status; raw tool: set_recording.

See RECORDING.md for the full flow: enable/disable, turn folder contents, replay via replay_trajectory, and the element_index doesn't-survive-across-sessions caveat.

Common error patterns