Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› eacl-reproducibility
eacl-reproducibility
GitHub用于审计EACL论文是否符合负责任NLP清单，核查超参数、算力、提示词、数据污染及多语言覆盖等声明的一致性。重点验证结果可复现性，识别误导性答案以防范桌拒风险，确保论文内容与清单回答严格对应。
Trigger Scenarios
运行EACL论文的负责任NLP清单审计
检查论文声明与清单答案的一致性
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill eacl-reproducibility -g -y
SKILL.md
Frontmatter
{
"name": "eacl-reproducibility",
"description": "Use when running the Responsible NLP checklist as a claims audit for an EACL paper, covering hyperparameter and compute disclosure, verbatim prompt and decoding reporting, data-contamination stance, variance and significance reporting, multilingual coverage claims, and consistency between the checklist answers and what the paper actually contains."
}
EACL Reproducibility
Use this to audit an EACL paper against the Responsible NLP checklist, which ARR files at
submission and which the action editor and reviewers read alongside the PDF. The checklist is
not paperwork: misleading answers are desk-rejection grounds, and inconsistencies between
the checklist and the paper are what careful EACL reviewers hunt for. Reopen the current
checklist at aclrollingreview.org/responsibleNLPresearch before auditing.
Treat the checklist as a claims audit
Every "yes" in the checklist implies a location in the paper. Walk the paper claim by claim and confirm each is backed:
| Claim type | Must disclose | Common EACL failure |
|---|---|---|
| Model results | Hyperparameters, tuning, model size | "Default settings" with no numbers |
| Compute | Hardware, run time, total budget | Silent on cost of large runs |
| LLM prompts | Verbatim prompts + decoding params | Paraphrased or omitted prompts |
| Data | Source, license, splits, preprocessing | Undocumented or "on request" data |
| Metrics | Variance over seeds / significance test | Single-run deltas reported as wins |
| Multilingual | Languages, resource levels, per-language results | Aggregate hides where it fails |
Contamination stance (LLM era)
- State explicitly whether evaluation data could have leaked into training — for closed LLMs
this is often unknowable, and the honest move is to say so and bound the risk rather than
claim clean evaluation. The "Leak, Cheat, Repeat" exemplar in
../../resources/exemplars/library.mdis the reference discipline. - Where feasible, run an overlap or decontamination check and report it.
Variance and significance floor
Reporting rule of thumb:
- >= 3-5 seeds for any headline comparison
- report mean +/- CI or std, not a single run
- a significance test when two systems are "close"
- never claim a win on an unreplicated single-run delta
Multilingual coverage honesty
- If the paper claims a cross-lingual or multilingual result, the checklist audit must confirm the languages are named, the resource levels are stated, and per-language results exist somewhere — an aggregate average is not evidence for every language.
- For lower-resourced languages, confirm dataset provenance and annotation context are documented; this is a recurring EACL reviewer expectation.
Checklist-to-paper consistency sweep
[ ] Every checklist "yes" maps to a section/appendix number
[ ] Hyperparameters + search space stated
[ ] Compute budget stated for expensive runs
[ ] Prompts + decoding params verbatim (if LLMs used)
[ ] Data source, license, splits, preprocessing documented
[ ] Variance/significance reported for headline claims
[ ] Contamination risk addressed honestly
[ ] Per-language results present for multilingual claims
[ ] AI-assistance use disclosed truthfully
Output format
[Reproducibility risk] Low / Medium / High
[Checklist-paper mismatches] <specific "yes" answers not backed in text>
[Disclosure gaps] <hyperparameters/compute/prompts/data>
[Evidence floor] <seeds/variance/significance findings>
[Contamination] <stance + any check run>
[Fix order] <what to add before the cycle deadline>
Version History
- 9f86f09 Current 2026-07-19 15:14
