Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› cogpsych-study-design
cogpsych-study-design
GitHub用于设计认知心理学实验，重点控制混淆变量、区分竞争模型并确保关键对比的统计功效。涵盖刺激构建、平衡方案及多实验逻辑，旨在强化研究设计的严谨性与推断力，不生成分析代码。
触发场景
设计单个或多个关联实验
构建刺激材料或平衡方案
回应关于混淆变量或统计功效的审稿质疑
为关键对比效应论证样本量
安装
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill cogpsych-study-design -g -y
SKILL.md
Frontmatter
{
"name": "cogpsych-study-design",
"description": "Use when designing the experiments for a Cognitive Psychology (Elsevier) manuscript so they tightly control confounds, discriminate competing models, and have adequate power across a multi-experiment program. Hardens stimulus construction, counterbalancing, design logic, and sample-size justification; it does not write analysis or modeling code."
}
Experiment Design (cogpsych-study-design)
Cognitive Psychology expects tightly controlled cognitive experiments whose design is engineered to
discriminate models, organized as a multi-experiment program in which each experiment adds
inference. The craft is in stimulus construction, counterbalancing, confound control, and powering the
critical contrast — not just the main effect. Co-design the experiments with the model
(cogpsych-theory-and-hypotheses).
When to trigger
- Designing an experiment or a multi-experiment series
- Constructing stimuli, item sets, and counterbalancing schemes
- A reviewer questioned confounds, power, design logic, or whether the design discriminates the models
- Justifying sample size for the critical contrast (often an interaction)
Design standards
- Design for discrimination. Build the design so the data produce the signature that separates the models (e.g., a manipulation that the rival accounts predict to diverge). A design that both models predict equally well wastes the experiment.
- Control researcher and stimulus degrees of freedom. Counterbalance condition/item assignment; control low-level confounds (frequency, length, familiarity, response mapping); randomize order; use attention/manipulation checks. Document the full stimulus pool, not a curated subset.
- Power the critical contrast. Justify N (and trials per cell) for the discriminating effect — often an interaction or a model parameter — not the easy main effect. State the assumed effect size and its source. Trials-per-participant is part of power for within-subjects designs.
- Multi-experiment logic. Say what each experiment adds: rules out a confound, extends scope, replicates the critical pattern, or tests a further model prediction. Avoid near-duplicate runs.
- Validity. Argue construct validity (does the task measure the process the model is about) and the generality of the claim across the stimulus space and population.
Powering the critical contrast — worked example (illustrative)
For the recognition-memory program, power the z-ROC shape contrast, not just overall accuracy.
Critical contrast: the diagnostic difference in z-ROC curvature between
UVSD and DPSD predictions.
Within-subjects: trials per participant drive ROC precision — target enough
old/new trials per confidence bin to estimate the slope reliably
(state the per-bin minimum, not just N).
Sample size: justified by simulation under each model (generate data from
UVSD and DPSD at plausible parameters; find N + trials at which
the model-recovery rate exceeds the target).
Across experiments: Exp 1 establishes the pattern; Exp 2 rules out a list-
composition confound; Exp 3 tests a further divergent prediction.
Stopping rule: fixed N + fixed trials; no optional stopping.
Justify sample size by model/parameter recovery simulation where the contrast is a model parameter, not only by a textbook power formula for a mean difference — this is the venue-appropriate move.
Pre-data lockdown checklist
| Degree of freedom | Lock before data? | Where it lives |
|---|---|---|
| Hypotheses + discriminating prediction | yes | preregistration / analysis plan |
| Models to be fit + comparison criteria | yes | analysis plan |
| Full stimulus pool + counterbalancing | yes | materials deposit |
| Trials per cell / per confidence bin | yes | design + power justification |
| Exclusion rules (RT, accuracy, dropout) | yes | preregistration |
| Stopping rule | yes | analysis plan |
| Exploratory analyses / model exploration | allowed, labeled | reported separately |
Design-stage reviewer pushback and the venue fix
- "Both models predict this design equally" → redesign so a manipulation makes the model predictions diverge; the signature must be diagnostic.
- "Possible stimulus confound (frequency/length)" → control or counterbalance it; report the matched pools; this objection lands hard here.
- "Underpowered for the interaction / too few trials" → power the critical contrast via simulation; report trials per cell, not only N.
- "Three near-identical experiments" → make each add inference (confound control, scope, further prediction).
Execution bridge (StatsPAI / Stata MCP)
Estimate and audit the design, don't only describe it. Full map:
execution-with-mcp. Cognitive Psychology is experimental — within-subject designs and mixed models dominate; report the model, the effect size, and multiple-comparison control.
detect_design→recommend→ fit withas_handle=true→audit_result.- Observational causal claims: staggered DiD (
callaway_santanna/sun_abraham+bacon_decomposition+honest_did_from_result); IV (effective_f_test+anderson_rubin_ci); RDD (rdrobust+mccrary_test). - Experiments: randomization-based inference,
romano_wolffor many-outcome family-wise control, andmediatefor mediation (not naive controlling-away). - Sensitivity:
oster_delta/sensemakrfor observational claims.
Report the effect size in interpretable units; route the full battery to the appendix/supplement. A run end-to-end (synthetic data, real returns) is in the JF execution walkthrough.
Anti-patterns
- A design that both rival models predict equally well (non-diagnostic)
- Uncontrolled low-level stimulus confounds or a curated stimulus subset
- Powering the easy main effect while the critical interaction/parameter is underpowered
- Too few trials per condition to estimate the model quantities reliably
- A multi-experiment paper of near-duplicate runs with no added inference
Output format
【Discrimination】does the design produce the model-separating signature? [Y/N]
【Confound control】counterbalancing + low-level controls + checks? [Y/N]
【Power】N + trials/cell justified for the critical contrast (simulation)? [Y/N]
【Degrees of freedom】stimuli, models, exclusions, stopping fixed in advance? [Y/N]
【Multi-experiment logic】what each experiment adds
【Next】cogpsych-data-analysis
Supplementary resources
../../resources/external_tools.md— stimulus tools, power/recovery simulation, preregistration templates../../resources/official-source-map.md— design and reporting expectations
版本历史
- 1839142 当前 2026-07-05 12:37
