ai-eval-plan

GitHub

为LLM或AI功能设计评估计划,将模糊的质量目标转化为可重复的测试流程。涵盖任务定义、数据集构建、多维度指标与评分标准、基线对比、通过阈值及CI回归门禁,防止质量退化发布。

exports/openclaw/ai-eval-plan/SKILL.md mohitagw15856/pm-claude-skills

Trigger Scenarios

如何评估提示词/模型/智能体 设置评估框架 定义AI功能的质量指标 构建回归门禁

Install

npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -g -y
More Options

Non-standard path

npx skills add https://github.com/mohitagw15856/pm-claude-skills/tree/main/exports/openclaw/ai-eval-plan -g -y

Use without installing

npx skills use mohitagw15856/pm-claude-skills@ai-eval-plan

指定 Agent (Claude Code)

npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a claude-code -g -y

安装 repo 全部 skill

npx skills add mohitagw15856/pm-claude-skills --all -g -y

预览 repo 内 skill

npx skills add mohitagw15856/pm-claude-skills --list

SKILL.md

Frontmatter
{
    "name": "ai-eval-plan",
    "homepage": "https:\/\/mohitagw15856.github.io\/pm-claude-skills\/skill\/ai-eval-plan.html",
    "metadata": {
        "openclaw": {
            "emoji": "🤖"
        }
    },
    "description": "Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt\/model\/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate."
}

AI Eval Plan Skill

You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship.

Required Inputs

Ask for these only if they aren't already provided:

  • The feature & task — what the model does and what "good output" means to a user.
  • Failure modes that matter — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow).
  • Available data — any real examples, logs, or labelled cases; or note there are none yet.
  • Who judges quality — automated checks, an LLM judge, human raters, or a mix.
  • The decision this gates — ship/no-ship, model selection, or prompt iteration.

Output Format

Eval Plan: [feature]

1. What we're measuring — the task, and a one-line definition of a good vs. bad response.

2. Eval dataset

  • Cases: how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set).
  • Coverage: the slices/scenarios that must be represented (edge cases, adversarial, each major input type).
  • Golden answers / references: present or not, and how they were created.

3. Metrics & rubric

  • Per-dimension scores — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes.
  • Automated checks — deterministic assertions first (valid JSON, contains required fields, no PII, latency budget).
  • LLM-as-judge — the judge prompt, the rubric it applies, and how you guard against its bias (calibrate against human labels on a sample).
  • Human eval — when it's required (safety, subjective quality) and the rater instructions.

4. Baselines — what each candidate is compared against (current prompt, previous model, a plain-prompt control).

5. The bar — the explicit threshold to ship (e.g. "≥4.2 avg correctness, 0 safety failures, p95 < 3s") and what happens if it's missed.

6. Regression gate — how this runs in CI on every change, and the score-drop threshold that blocks a merge.

Quality Checks

  • Each metric has an explicit rubric with anchors — not just a name
  • Deterministic/automated checks are used wherever possible before reaching for an LLM judge
  • The LLM judge is calibrated against human labels on at least a sample
  • The eval set includes adversarial and edge cases, not just happy-path examples
  • There is a single, explicit numeric bar for the ship decision
  • The plan specifies how it runs as a regression gate, not just a one-time check

Anti-Patterns

  • Do not rely on a single overall score — a feature can pass on average while failing every safety case
  • Do not trust an LLM judge you haven't calibrated against humans — it has its own blind spots and biases
  • Do not eval only on happy-path inputs — the failures live in the edges and the adversarial cases
  • Do not let the eval set leak into the prompt/few-shot examples — that's training on the test set
  • Do not define the pass bar after seeing the scores — set the threshold before you run, or it means nothing

Based On

LLM evaluation practice — task-grounded rubrics, LLM-as-judge with human calibration, and regression-gated CI evals.

Version History

  • 54fad50 Current 2026-07-19 12:10

Same Skill Collection

exports/openclaw/360-feedback-template/SKILL.md
exports/openclaw/401k-plan-decoder/SKILL.md
exports/openclaw/ab-test-planner/SKILL.md
exports/openclaw/ab-test-readout/SKILL.md
exports/openclaw/accessibility-audit/SKILL.md
exports/openclaw/account-plan/SKILL.md
exports/openclaw/acquirer-red-team/SKILL.md
exports/openclaw/ad-copy/SKILL.md
exports/openclaw/aeo-optimizer/SKILL.md
exports/openclaw/agenda-or-cancel/SKILL.md
exports/openclaw/agent-design-review/SKILL.md
exports/openclaw/agent-observability-spec/SKILL.md
exports/openclaw/agent-spec/SKILL.md
exports/openclaw/ai-ethics-review/SKILL.md
exports/openclaw/ai-feature-prd/SKILL.md
exports/openclaw/ai-product-canvas/SKILL.md
exports/openclaw/air-quality/SKILL.md
exports/openclaw/altitude-shifter/SKILL.md
exports/openclaw/ambiguity-resolver/SKILL.md
exports/openclaw/analyst-relations-brief/SKILL.md
exports/openclaw/announcement-card/SKILL.md
exports/openclaw/api-docs-writer/SKILL.md
exports/openclaw/api-test-plan/SKILL.md
exports/openclaw/api-versioning-strategy/SKILL.md
exports/openclaw/apology-letter/SKILL.md
exports/openclaw/architecture-decision-record/SKILL.md
exports/openclaw/architecture-diagram/SKILL.md
exports/openclaw/archive-strategy/SKILL.md
exports/openclaw/assumption-bounty/SKILL.md
exports/openclaw/assumption-mapper/SKILL.md
exports/openclaw/async-update-format/SKILL.md
exports/openclaw/auto-repair-estimate-decoder/SKILL.md
exports/openclaw/autopilot-charter/SKILL.md
exports/openclaw/benefits-decoder/SKILL.md
exports/openclaw/bid-tender-review/SKILL.md
exports/openclaw/board-deck-narrative/SKILL.md
exports/openclaw/board-minutes/SKILL.md
exports/openclaw/board-pre-read/SKILL.md
exports/openclaw/bom-cost-review/SKILL.md
exports/openclaw/bookkeeping-categorization/SKILL.md
exports/openclaw/boolean-search-builder/SKILL.md
exports/openclaw/brag-doc/SKILL.md
exports/openclaw/brainstorming/SKILL.md
exports/openclaw/brief-builder/SKILL.md
exports/openclaw/briefing-note/SKILL.md
exports/openclaw/budget-builder/SKILL.md
exports/openclaw/budget-variance-analysis/SKILL.md
exports/openclaw/bug-diagnosis/SKILL.md
exports/openclaw/bug-report/SKILL.md

Metadata

Files
0
Version
471c606
Hash
736b1aca
Indexed
2026-07-19 12:10

- 위키
Copyright © 2011-2026 iteam. Current version is 2.155.2. UTC+08:00, 2026-07-30 23:46
浙ICP备14020137号-1 $방문자$