Agent Skills
› flutter/agent-plugins
› run-evals
run-evals
GitHub用于运行技能评估、基准测试及性能对比。通过读取配置、调度子代理执行任务并验证输出,最后根据标准进行评分,以检验技能表现。
Trigger Scenarios
用户要求运行评估(evals)
测试技能性能
运行基准测试
对比基线与启用技能后的执行效果
Install
npx skills add flutter/agent-plugins --skill run-evals -g -y
SKILL.md
Frontmatter
{
"name": "run-evals",
"metadata": {
"internal": true
},
"description": "Run evaluations for one, multiple, or all skills using the agent orchestration framework. Make sure to use this skill whenever the user asks to run evals, test a skill's performance, run benchmarks, or compare baseline versus with-skill execution."
}
Run Skill Evals
- Read Framework: Read
<target-package-root>/evals/README.mdfor understanding the difference between per-skill evals and cross-skill evals (where<target-package-root>is the directory containing the.agentsorskillsfolder). - Locate Targets: Find target
evals/evals.jsonfiles inside.agents/skills/and/orskills/. For cross-skill evaluations, look for*_evals.jsonfiles directly in<target-package-root>/evals/. - Determine Agent Configuration: Check the
agent_configfield in the target target JSON file to determine the environment/harness to spawn. Ifagent_configis"bare-agent", spawn a subagent with thebare-agentprofile. If it is a specific contributor profile (e.g."reidbaker-agent"), use that profile to provide the necessary contributor context. - Orchestrate: By default, run an Integration Test by spawning a single With-Skill subagent using
Workspace: branchand the identifiedagent_config.- Provide the task prompt. See
resources/with_skill_execution_prompt.mdfor the template. When filling in<path-to-skill>, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in<path-to-skill>with"none (cross-skill meta-eval)". Also, replace<target-package-root>with the actual directory path in both templates. - Only if the user explicitly requests a comparison or benchmark, also spawn a Baseline subagent. See
resources/baseline_execution_prompt.mdfor the template. Instruct the subagent(s) to return theirgit diffand verification outputs (dart pub get,dart format,dart analyze,dart test) without committing. Ensure you instruct them to run these commands exclusively from within the<target-package-root>directory to avoid analyzing unrelated packages. CRITICAL: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace. WORKSPACE LIMITATION WARNING: If the user has multiple active workspaces mounted, theWorkspace: branchfeature will fail. In this situation, you MUST warn the user that running concurrent evaluations inWorkspace: inheritmode will cause git state bleed and cross-eval pollution (e.g., changes made by a failure scenario will be visible to a success scenario running simultaneously in the same shared directory). Instruct the user to fix this by closing all workspaces except the primary package workspace, and then re-run the evaluations. Do NOT silently fallback toWorkspace: inheritfor concurrent tasks.
- Provide the task prompt. See
- Grade: Parse the combined rubric (resolving
repo_criteria+evals.jsonexpectations). Use the grading instructions inresources/agent_judge_prompt.md. When an expectation fails, you MUST explicitly list both the expectation and what was actually found that caused the failure. - Artifact: Grade the outputs and generate a Markdown artifact (e.g.,
<skill>_eval_results.md) containing the metadata, pass/fail rationale, and raw diffs/stdout.
Version History
- c3c71cf Current 2026-08-20 07:12


