Agent Skills
› langchain-ai/design-agent
› design-agent-evals
design-agent-evals
GitHub用于评估设计代理变更效果,通过运行前后对比测试、分析报告及将真实故障转化为测试任务,量化模型表现与成本。
Trigger Scenarios
评估指令、技能或模型的变更
询问代理性能如何时
发现真实请求失败并需添加测试用例时
Install
npx skills add langchain-ai/design-agent --skill design-agent-evals -g -y
SKILL.md
Frontmatter
{
"name": "design-agent-evals",
"description": "Find out whether a change made Design Agent better or worse. Run the eval set or a few tasks before and after against the same model, read the reports, look at the files, and turn a real request that went wrong into a task. Use after changing instructions, skills, checks or models, or when someone asks how good it is."
}
Evaluate a change
Eval runs spend model and media money. eval run prints the budget: show it to the person and get
a yes. The whole set costs about $120 per model; a few tasks cost a few dollars.
uv run design-agent eval listshows every task with its dollar and minute caps.- Pick the tasks the change touches, plus one behavior task: still tasks for a stills skill,
the teaser and film for motion,
exact-copyfor copy rules. - Run before and after, same model, same tasks:
uv run design-agent eval run --task <id> --task <id> --out workspace/out/evals/<label>. - Read the reports. Each folder has
report.json,README.md(failed checks per task, spend, time) andfiles/<task>/. Open the files and look at them. The graders check outputs, sizes, copy and behavior, never taste: a person judges taste. - Report numbers from the reports: checks passed, spend and time, before against after. Do not summarize a run you did not read.
- Turn a real failure into a task. When a thread shows something going wrong, add an entry to
tests/eval/tasks.jsonmodelled on a similar task (the request, expected outputs and sizes, exact copy lines, budgets), thenuv run pytest -q tests/unit/test_evals.py.
Graders and reports: Evals.
Version History
- 8134d7c Current 2026-11-20 21:40


