a-b-test-design
GitHub指导设计严谨的A/B测试，包括构建假设、控制变量、选择核心指标及计算样本量。适用于需量化评估的大规模变更场景，提供结构化的实验设计框架与最佳实践。
Trigger Scenarios
Install
npx skills add Owl-Listener/designer-skills --skill a-b-test-design -g -y
SKILL.md
Frontmatter
{
"name": "a-b-test-design",
"description": "Design an A\/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`."
}
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
Version History
-
20e34c4
Current 2026-08-19 23:22
优化技能描述，增加使用触发条件和边界说明，解决与其他技能的重叠问题，确保描述包含'Use when'句式并限制长度。
- acc3e57 2026-07-25 05:16
