Agent Skills › A-EVO-Lab/a-evolve › screenshot-analysis

screenshot-analysis

GitHub

指导如何结合截图与无障碍树分析GUI元素，通过坐标映射实现自动化点击、输入等交互操作，适用于桌面及Web应用测试。

seed_workspaces/osworld/skills/screenshot-analysis/SKILL.md A-EVO-Lab/a-evolve

Trigger Scenarios

需要模拟鼠标点击或键盘输入以操作图形界面需要解析屏幕元素状态以验证UI交互结果

Install

npx skills add A-EVO-Lab/a-evolve --skill screenshot-analysis -g -y

More Options

Non-standard path

npx skills add https://github.com/A-EVO-Lab/a-evolve/tree/main/seed_workspaces/osworld/skills/screenshot-analysis -g -y

Use without installing

npx skills use A-EVO-Lab/a-evolve@screenshot-analysis

指定 Agent (Claude Code)

npx skills add A-EVO-Lab/a-evolve --skill screenshot-analysis -a claude-code -g -y

安装 repo 全部 skill

npx skills add A-EVO-Lab/a-evolve --all -g -y

预览 repo 内 skill

npx skills add A-EVO-Lab/a-evolve --list

SKILL.md

Frontmatter

{
    "name": "screenshot-analysis",
    "description": "How to interpret accessibility tree elements and correlate them with screenshot regions for accurate GUI interaction."
}

Screenshot & Accessibility Tree Analysis

Reading the Accessibility Tree

The a11y_tree is your primary guide for finding interactive elements:

[role=window] "Document - LibreOffice Calc"
  [role=menubar]
    [role=menuitem] "File"
    [role=menuitem] "Edit"
  [role=toolbar]
    [role=button] "Save" (x=120, y=45)
  [role=table] "Sheet"
    [role=cell] "A1" (x=80, y=200)

Coordinate mapping

a11y_tree provides (x, y) center coordinates for each element
Use these for pyautogui.click(x, y)
If coordinates seem wrong, the element may be scrolled off-screen

Element roles cheat sheet

button: clickable action (Save, OK, Cancel)
menuitem: menu entry (click to open submenu or execute)
textbox/text: editable text field (click then type)
combobox: dropdown (click to open, then select)
checkbox: toggle (click to check/uncheck)
cell: spreadsheet cell (click to select, then type)
tab: tab selector (click to switch)
dialog: modal window (must handle before other actions)

When a11y_tree is insufficient

Some custom widgets don't expose a11y info — use screenshot + coordinates
For canvas-based apps (GIMP): rely on menu navigation, not direct canvas clicks
For web content in Chrome: a11y_tree reflects DOM, usually reliable

Tips

Always check a11y_tree AFTER an action to verify the state changed
If an element disappears from a11y_tree, the UI updated (menu closed, dialog dismissed)
Count elements to verify you're clicking the right one (e.g., 3rd button in toolbar)