evals
GitHub用于创建和运行 Agent 行为评估（Evals）。支持通过命令或并行子代理执行测试，依据预期标准判断成功或失败，并输出标准化结果供 CI 解析。同时提供创建新 Evals 的功能，强调自包含和可复现性。
Trigger Scenarios
Install
npx skills add NeverSight/learn-skills.dev --skill evals -g -y
SKILL.md
Frontmatter
{
"name": "evals",
"description": "Run and create evals for testing agent behavior. Use when the user wants to create or run an eval."
}
Eval Skill
Run and create evals for testing agent behavior.
Discovering Evals
Evals are markdown files matching *.eval.md. Use glob to find them:
**/*.eval.md
A common pattern is to collect evals in an evals/ directory.
Eval Structure
An eval file contains a prompt and an expectation:
# Eval Title
<prompt>
Instructions for the agent to execute.
</prompt>
<expectation>
Success criteria - describe what must be true for the eval to pass.
</expectation>
Running an Eval
- Read the eval file
- Extract the
<prompt>content - Spawn a subagent with the prompt (runs in current working directory with shared state)
- The subagent evaluates its own result against the
<expectation>using LLM judgment - Subagent outputs
SUCCESSorFAILwith reasoning
When running multiple evals, spawn all subagents in parallel. Report aggregate results at the end.
Always end output with exactly one of these lines for CI parsing:
eval result: pass— all evals passedeval result: fail— one or more evals failed
Subagent Instructions
IMPORTANT: The subagent must only test and observe. It must NOT attempt to fix, modify, or change anything to make the expectation pass. The subagent executes the prompt, observes the outcome, and reports whether the expectation was met. If the expectation fails, report FAIL — do not try to make it pass.
Commands
Run a single eval:
/eval run <path-to-eval.eval.md>
Run all evals:
/eval run-all
Creating an Eval
Gather from the user:
- Context - The process or flow to evaluate
- Expectation - Success criteria in natural language
/eval create <name>
Write the eval to <name>.eval.md in the current directory.
Isolation
When creating an eval, try to make it self-contained and reproducible. This isn't critical, but helps:
- Try to avoid hardcoded paths — prefer relative paths or have the prompt create its own working directory rather than encoding specific temp directories or absolute paths.
- Try to avoid external state — if the process relied on existing files or services, consider whether the eval should create that state itself.
- Parameterize where possible — replace specific values (ports, filenames, IDs) with generic ones the eval can generate.
If you see an opportunity to improve isolation but need clarification, ask the user.
Version History
- e0220ca Current 2026-07-05 23:30
