Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› eacl-experiments
eacl-experiments
GitHub指导设计或审计EACL论文的实证证据，确保公平基线、范围匹配声明、显著性检验、数据污染控制及人类评估规范，以生成严谨可复现的实验结果。
触发场景
设计EACL论文实验方案
审计NLP论文实证证据
检查基线公平性与数据污染
安装
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill eacl-experiments -g -y
SKILL.md
Frontmatter
{
"name": "eacl-experiments",
"description": "Use when designing or auditing the empirical evidence for an EACL paper, covering tuned and LLM baselines, multilingual breadth matched to the claim, significance and variance floors, human-evaluation agreement, data-contamination controls, ablations, and error taxonomies, so that every stated result is measured rather than asserted."
}
EACL Experiments
Use this to make an EACL paper's evidence hold up under NLP review. EACL rewards well-scoped
questions answered with careful controls over leaderboard maximalism — its best papers include
analyses and critiques, not only state-of-the-art systems (see
../../resources/exemplars/library.md). Design the evidence to match the claim exactly, no
broader.
Baselines that make a comparison fair
- Include a tuned baseline, not a strawman: an under-tuned competitor makes a win meaningless. State the search space for both your method and the baselines.
- For LLM-based work, include the obvious prompt/few-shot baseline and report its prompts and decoding settings; a gain over an unreported baseline is not credible.
Match breadth to the claim
| Claim | Required breadth |
|---|---|
| "Works for language L" | Solid results on L, honestly scoped |
| "Cross-lingual / multilingual" | Enough languages across resource levels; per-language results |
| "General method" | Multiple tasks/datasets, not one convenient benchmark |
| "Robust" | Stress tests / shifts, not just in-distribution |
A multilingual claim backed by two high-resource languages is the classic EACL over-reach — the morphology-across-57-languages exemplar shows the bar.
Significance and variance floor
Evidence floor for a headline comparison:
seeds: >= 3-5 runs
report: mean +/- CI (or std), never a lone run
significance: a test when systems are close
ablations: isolate each component's contribution
Contamination controls
- For any benchmark evaluated with LLMs, address whether the test data could have leaked. Report an overlap/decontamination check where feasible, or bound the risk honestly for closed models. This is a live EACL concern, not a formality.
Human evaluation done properly
- If human judgments are a result, report the number of annotators, guidelines, pay, and inter-annotator agreement — an unmeasured human eval is a soft target for reviewers.
- Release the annotation materials (see
eacl-artifact-evaluation).
Error analysis as a first-class result
- A quantified error taxonomy ("X% agreement errors, Y% named-entity errors, examples in Table N") often carries more scientific weight than another decimal of accuracy, and plays to EACL's analysis-friendly reviewing.
Audit checklist
[ ] Baselines tuned, search spaces stated
[ ] LLM baselines with verbatim prompts + decoding
[ ] Breadth matches the claim (per-language results if multilingual)
[ ] >= 3-5 seeds; variance/CIs reported
[ ] Significance test where systems are close
[ ] Ablations isolate each component
[ ] Contamination addressed
[ ] Human eval: annotators, agreement, pay reported
[ ] Error analysis quantified
Output format
[Evidence strength] Strong / Adequate / Weak
[Baseline fairness] <tuned? LLM baseline reported?>
[Breadth vs claim] <matched / over-reaching>
[Variance + significance] <seeds, CIs, tests>
[Contamination + human eval] <controls present?>
[Fix order] <experiments to add/scope before the cycle deadline>
版本历史
- 9f86f09 当前 2026-07-19 15:14
