Agent Skills
› NeverSight/learn-skills.dev
› macpilot-ui-inspector
macpilot-ui-inspector
GitHub通过MacPilot的AX API检查和交互macOS应用UI元素。支持查找、点击、读写控件值、获取层级结构及快捷键，实现精准的程序化界面控制。
Trigger Scenarios
查找特定按钮或文本字段
读取UI控件的值或状态
根据标签点击按钮
检查应用完整UI层级
在文本框中设置值或切换复选框
发现应用支持的键盘快捷键
识别特定屏幕坐标处的元素
Install
npx skills add NeverSight/learn-skills.dev --skill macpilot-ui-inspector -g -y
SKILL.md
Frontmatter
{
"name": "macpilot-ui-inspector",
"description": "Inspect and interact with macOS UI elements using MacPilot accessibility APIs. Find buttons, text fields, labels, and other elements by role, label, or position, then click, read, or modify them."
}
MacPilot UI Inspector
Use MacPilot's accessibility (AX) commands to inspect, find, and interact with UI elements in any macOS application. This skill enables precise programmatic control of app interfaces.
When to Use
Use this skill when the user asks to:
- Find a specific button, text field, checkbox, or UI element in an app
- Read the value or state of a UI control
- Click a button or interact with a control by its label
- Inspect the full UI hierarchy/accessibility tree of an app
- Set values in text fields or toggle checkboxes programmatically
- Discover what keyboard shortcuts an app supports
- Identify elements at specific screen coordinates
Commands
List UI Elements
macpilot ui list --app "Safari" --json # All elements
macpilot ui list --app "Safari" --role AXButton --json # Only buttons
macpilot ui list --app "Safari" --depth 5 --json # Deeper scan
macpilot ui list --app "Safari" --hierarchy --json # With hierarchy
Find Elements by Text
macpilot ui find "Save" --app "TextEdit" --json # Find by label
macpilot ui find "Save" --role AXButton --json # Filter by role
macpilot ui find "Save" --exact --json # Exact match only
Find Element with Coordinates
macpilot ui find-text "Submit" --app "Safari" --json
# Returns: position {x, y}, size {w, h} - use for clicking
Click UI Element by Label
macpilot ui click "Save" --app "TextEdit" --json
macpilot ui click "Cancel" --role AXButton --json
Get/Set Values
macpilot ui get-value "Search" --app "Safari" --json # Read field value
macpilot ui set-value "Search" "query text" --app "Safari" # Set field value
macpilot ui set-value "Dark Mode" "1" --role AXCheckBox # Toggle checkbox
Set Focus
macpilot ui set-focus "Search" --app "Safari" --json # Focus an element
Scroll Within Elements
macpilot ui scroll "content" down 5 --app "Safari" # Scroll element
Inspect Element Attributes
macpilot ui attributes "Save" --app "TextEdit" --json # All AX attributes
Elements at Coordinates
macpilot ui elements-at 500 300 --json # What's at x=500 y=300
macpilot ui elements-at 500 300 --radius 50 --json # Search wider area
Accessibility Tree
macpilot ui tree --app "Finder" --json # Full AX tree
macpilot ui tree --app "Finder" --depth 3 --json # Limit depth
Keyboard Shortcuts
macpilot ui shortcuts --app "Safari" --json # All shortcuts
macpilot ui shortcuts --app "Safari" --menu File --json # Menu-specific
Wait for Elements
macpilot wait element "Download Complete" --app "Safari" --timeout 30 --json
Common AX Roles
| Role | Description |
|---|---|
AXButton |
Buttons (push, toggle) |
AXTextField |
Text input fields |
AXTextArea |
Multi-line text areas |
AXStaticText |
Labels and display text |
AXCheckBox |
Checkboxes and toggles |
AXRadioButton |
Radio buttons |
AXPopUpButton |
Dropdown menus |
AXComboBox |
Combo boxes |
AXTable |
Tables and lists |
AXRow |
Table/list rows |
AXMenuItem |
Menu items |
AXToolbar |
Toolbars |
AXScrollArea |
Scrollable regions |
AXWindow |
Windows |
AXSheet |
Modal sheets |
AXImage |
Images |
AXLink |
Hyperlinks |
AXGroup |
Generic container |
Workflow Patterns
Pattern 1: Find and Click
# Always focus app first
macpilot app focus "Safari"
# Find the element to verify it exists
macpilot ui find "Downloads" --app "Safari" --role AXButton --json
# Click it
macpilot ui click "Downloads" --app "Safari"
Pattern 2: Read Form State
macpilot app focus "System Settings"
macpilot ui get-value "Computer Name" --app "System Settings" --json
Pattern 3: Fill a Form
macpilot app focus "MyApp"
macpilot ui set-value "Name" "John Doe" --app "MyApp"
macpilot ui set-value "Email" "john@example.com" --app "MyApp"
macpilot ui click "Submit" --app "MyApp"
Pattern 4: Explore Unknown UI
# Start with a broad scan
macpilot ui tree --app "SomeApp" --depth 2 --json
# Narrow down to specific elements
macpilot ui list --app "SomeApp" --role AXButton --json
# Inspect a specific element
macpilot ui attributes "Settings" --app "SomeApp" --json
Pattern 5: Click by Coordinates (Fallback)
# When label-based clicking fails, find coordinates first
macpilot ui find-text "Submit" --app "MyApp" --json
# Use returned x,y to click
macpilot click 450 320
Tips
- Always call
macpilot app focusbefore any UI interaction - Use
--jsonoutput for reliable parsing of element properties - When
ui clickfails, fall back toui find-text+macpilot click x y - Use
ui tree --depth 2first to understand app structure before deep scans - AX permission must be granted to MacPilot.app in System Settings
- The
--rolefilter significantly speeds up element searches in complex apps
Version History
- e0220ca Current 2026-07-05 23:21
