Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› revedres-comprehensiveness-and-balance
revedres-comprehensiveness-and-balance
GitHub用于教育研究综述或元分析的全面性与平衡性测试，验证覆盖度、偏倚风险及异质性。防止遗漏与脆弱性，强调证据加权而非投票计数，确保合成结果稳健可靠。
Trigger Scenarios
已有语料和框架但未检验覆盖度或稳健性
报告合并效应量但缺乏异质性、偏倚或敏感性分析
担心审稿人指出遗漏的研究或矛盾发现
仅对冲突研究进行计数而未按设计质量加权
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill revedres-comprehensiveness-and-balance -g -y
SKILL.md
Frontmatter
{
"name": "revedres-comprehensiveness-and-balance",
"description": "Use when testing exhaustiveness, risk-of-bias, heterogeneity, and sensitivity for a Review of Educational Research (RER) review or meta-analysis. Hardens the synthesis against omission and fragility; it does not build the conceptual spine (revedres-organizing-framework) or settle coding reliability and open materials (revedres-transparency-and-reproducibility)."
}
Comprehensiveness & Balance (revedres-comprehensiveness-and-balance)
When to trigger
- The corpus and framework exist, but you have not stress-tested coverage or robustness
- A pooled effect is reported without heterogeneity, bias, or sensitivity analysis
- You worry a reviewer will name an omitted study, school, or contradictory finding
- Conflicting studies are being tallied rather than weighed by design quality
Comprehensiveness: prove there are no holes
RER asks for comprehensive reviews; the burden is on you to make exhaustiveness provable, not asserted.
- Saturation evidence. Show that the search reached the point where new searches stopped yielding eligible studies (from
revedres-literature-synthesis). - Named-omission test. Could an informed reviewer name an important study, research group, or adjacent literature you left out? If yes, include it or justify the boundary explicitly.
- Grey-literature & language reach. Excluding dissertations, reports, or non-English work narrows scope and biases effects — acknowledge and, where feasible, widen.
Balance: weigh evidence, never vote-count
The amateur move is vote-counting — tallying significant vs. null studies. The RER standard is to weigh evidence by what each study measures and how credibly.
- Risk-of-bias appraisal. Apply the a-priori tool to every included study; let credibility, not count, drive emphasis. A handful of well-identified studies can outweigh many weak ones.
- Estimand discipline. Reconcile conflicting findings by asking whether studies estimate the same object (population, construct, horizon, comparison). Apparent contradictions often dissolve.
- Steelman rival perspectives. State each theoretical or methodological camp at its strongest before its limits; flag — and bracket — your own program so emphasis is identity-blind.
Robustness (meta-analysis): make the number survive scrutiny
A pooled effect is a claim, not a fact, until you show it is not an artifact.
| Probe | What it guards against |
|---|---|
| Heterogeneity (Q, I², τ²) | reporting one number for a mix of different effects |
| Moderator analysis | masking real variation the framework should explain |
| Publication-bias diagnostics (funnel, Egger, trim-and-fill, p-curve/selection models) | an inflated effect from missing null results |
| Sensitivity analysis (leave-one-out, influence, alternative models) | a result driven by one study or one modeling choice |
| Dependent-effects handling (multilevel / robust variance) | false precision from multiple effects per sample |
For a narrative synthesis, the analogues are: confidence in the body of evidence (e.g. a GRADE-style judgment), explicit handling of conflicting findings, and a sensitivity check on which conclusions survive dropping the weakest studies.
Checklist
- Saturation documented; no eligible study/database a reviewer could name as missing
- Grey-literature/language exclusions acknowledged with their bias implications
- Risk-of-bias appraised for every study; emphasis tracks credibility, not count
- Conflicting findings reconciled by estimand + design, not tallied
- Rival camps steelmanned; author's own work bracketed for identity-blind emphasis
- (Meta) heterogeneity, moderators, publication-bias, and sensitivity all reported
- (Meta) dependent effects modeled (multilevel / robust variance), not ignored
- (Narrative) strength-of-evidence judged and a drop-the-weakest sensitivity check run
Anti-patterns
- Vote-counting significant vs. null studies as if each carries equal weight
- A single pooled effect with no I²/τ², no moderators, and no publication-bias check
- Ignoring dependent effect sizes, manufacturing false precision
- Excluding grey literature silently, then reporting an upward-biased effect
- Caricaturing the camp the author disagrees with instead of steelmanning it
- Treating "I found a lot of studies" as proof of comprehensiveness without saturation evidence
Output format
【Saturation】documented? Y/N — named-omission test passed? Y/N
【Grey lit / language】exclusions + bias implication stated? Y/N
【Risk of bias】appraised for all studies; emphasis credibility-weighted? Y/N
【Conflict handling】reconciled by estimand/design (not vote-count)? Y/N
【Heterogeneity】I²/τ² + moderators reported? Y/N (meta) | strength-of-evidence judged (narrative)
【Publication bias】funnel/Egger/trim-fill/p-curve run? Y/N
【Sensitivity】leave-one-out / alt models / dependent-effects model? Y/N
【Next step】→ revedres-tables-figures (PRISMA flow, forest/funnel, coding tables)
Version History
- 1839142 Current 2026-07-05 14:22
