Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› ijoc-data-analysis
ijoc-data-analysis
GitHub用于运行、报告IJOC计算实验并组装可复现代码数据。确保结果可信，处理调优、随机性及基准偏差，执行统计检验（如性能曲线、Wilcoxon检验），并按要求整理GitHub仓库以符合期刊规范。
Trigger Scenarios
需要运行和解释计算实验结果时
审稿人质疑性能提升的真实性或存在调优/随机性偏差时
需要将原始输出转化为IJOC要求的统计比较分析时
正在组装并提交IJOC所需的GitHub软件和数据仓库时
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill ijoc-data-analysis -g -y
SKILL.md
Frontmatter
{
"name": "ijoc-data-analysis",
"description": "Use when running and reporting the computational experiments — and assembling the reproducible code\/data deposit — for an INFORMS Journal on Computing (IJOC) manuscript. Turns a designed protocol (see ijoc-methods) into defensible, reproducible results; it does not redesign the experiment."
}
Computational Experiments & Reproducibility (ijoc-data-analysis)
When to trigger
- The protocol is designed and you are now running experiments and interpreting results
- A referee questions whether a performance win is real versus tuning, seed luck, or benchmark selection
- You need to turn raw outputs into the statistical comparison IJOC expects (not just a table of best times)
- You are assembling the IJOC GitHub Software and Data Repository deposit and want it to reproduce the paper
Making the computational claim defensible
IJOC's distinctive risk is a result that is real on the page but an artifact underneath. Pre-empt the three ways a referee will attack it.
- Tuning artifact. Show the win holds with tuning done symmetrically on a disjoint set. Report performance at default and tuned for both your method and the baselines, so the gain is not hidden in hyperparameters.
- Seed/variance artifact. For any stochastic component, run multiple seeds (commonly ≥10) and report mean, dispersion (sd/IQR), and the distribution — a box/violin plot or a table of quartiles — not a single best run. State the seeds; they go in the deposit.
- Benchmark-selection artifact. Report on the full standard instance set, including instances where you lose or time out; aggregate with a performance profile (Dolan–Moré) so the whole picture is visible. Cherry-picking the favorable subset is the fastest way to lose a referee's trust.
The statistics IJOC reviewers expect
Means and "X is fastest" are not enough. Use the comparisons that suit computational data:
- Performance profiles to compare solvers across a whole instance set (fraction solved within a factor of the best).
- Pairwise nonparametric tests (Wilcoxon signed-rank for matched instances; sign test) rather than t-tests on heavy-tailed runtimes.
- Multiple-comparison control (e.g., Holm/Bonferroni) when comparing several methods at once.
- Scaling evidence: runtime/memory vs. instance size on log axes, with the empirical growth compared to the theory from
ijoc-theory-development. - Equal-effort comparisons: quality at equal time, or time at equal quality — never "ours had more time."
Report optimality gaps with their definition, dual bounds where available, and time limits explicitly (a "solved" count is meaningless without the limit).
Assembling the IJOC reproducibility deposit
Accepted IJOC papers deposit artifacts in the INFORMSJoC GitHub organization, and the deposit is part of the contribution — design it to reproduce the paper. Concretely (检索于 2026-06；以官网为准):
- Start from the template repository (
github.com/INFORMSJoC/2019.0000) and replace placeholders with your manuscript IDXXXX.YYYY. - Three root files are required:
README.md(with a## Citesection as the first subheading, using DOI10.1287/ijoc.XXXX.YYYY.cd),LICENSE, andAUTHORS. - Organize into
src/,data/(with documented provenance),scripts/(to replicate experiments),results/,docs/. - Pin dependencies (
requirements.txt,Manifest.toml, environment files). A best-faith reproducibility effort is expected even if exact bit-reproduction is not. - On acceptance, a tagged snapshot (
vXXXX.YYYY) is archived and a code DOI (….cd) is minted alongside the article DOI — so the deposited state must match the published results.
The README's replication section should let a reader regenerate each table/figure from scripts/. If your results/ directory maps one-to-one to the paper's exhibits, the reproducibility reviewer's job — and yours during the R&R — becomes mechanical.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. INFORMS JoC is computing / algorithms and methodology; the causal-inference chain below applies only to its empirical-evaluation papers, not to algorithm design.
- Many outcomes / specifications:
romano_wolf(step-down FWER) orbenjamini_hochberg— report the adjusted threshold. - OVB sensitivity:
oster_delta/sensemakr. - Inference:
wild_cluster_bootstrap(few clusters),twoway_cluster/conley; multilevel data → cluster at the right level. - Re-fit off one handle:
audit_result(result_id)lists the missing checks and the exactsuggest_functionfor each. - Exhibits:
etable/did_summary_to_latexfrom the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the appendix. See the executed chain in the JF execution walkthrough.
Checklist
- Win shown at default and tuned; tuning symmetric on a disjoint set
- ≥ multiple seeds for stochastic runs; dispersion and distribution reported; seeds recorded
- Full standard instance set reported, including losses/timeouts; performance profile shown
- Nonparametric pairwise test (Wilcoxon) + multiple-comparison control where needed
- Scaling plot vs. size on log axes; compared to the theoretical complexity
- Gaps defined; dual bounds and time limits stated
- Deposit built from the IJOC template: README(+
## Cite)/LICENSE/AUTHORS; src/data/scripts/results; pinned deps; seeds -
results/maps to the paper's tables/figures; scripts regenerate them
Anti-patterns
- A single best run reported for a randomized algorithm
- Means and "fastest" with no statistical test or performance profile
- Dropping the instances where the method loses or times out
- Reporting "solved 47/50" without the time limit
- A deposit that is a code dump with no README replication path, or that does not reproduce the published numbers
- Letting the deposited snapshot drift from the accepted results
Output format
【Journal】INFORMS Journal on Computing
【Skill】ijoc-data-analysis
【Headline result】the computational win, stated with its metric
【Artifact controls】default+tuned / #seeds+dispersion / full instance set
【Statistics】performance profile + Wilcoxon (+ MC control)? [Y/N]
【Scaling vs. theory】[consistent / discrepancy noted]
【Deposit】template/README+Cite/LICENSE/AUTHORS/scripts/seeds ready? [Y/N]
【Next skill】ijoc-contribution-framing
Version History
- 1839142 Current 2026-07-05 13:21
