Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› ase-experiments
ase-experiments
GitHub指导ASE论文实验设计，确保工具在真实系统上公平对比基线。涵盖按声明类型匹配证据、真实主体选择与溯源、可运行基线配置、消融实验隔离组件、明确预言机及防过拟合、统计效应量报告，以通过同行评审。
Trigger Scenarios
设计自动化软件工程论文的实验部分
审计ASE论文的实验评估有效性
检查基线对比的公平性
验证实验主体的真实性与溯源
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill ase-experiments -g -y
SKILL.md
Frontmatter
{
"name": "ase-experiments",
"description": "Use when designing or auditing the evaluation of an ASE (IEEE\/ACM Automated Software Engineering) paper, covering real subject systems, fair runnable tool baselines, task-matched effectiveness metrics, ablations that isolate a learned component, oracle and correctness validation, contamination-aware LLM handling, and provenance for mining."
}
ASE Experiments
Match the evidence to the automation's claim. ASE evaluations are judged on whether a tool or technique actually does what it claims on real subjects, compared fairly against the closest runnable automation. This is the axis reviewers weight most, and the one that most often becomes a Revision criterion.
Start from the claim shape
Different automations demand different evidence:
| Automation claim | Evidence that matches | Common failure |
|---|---|---|
| Detection (bugs, smells, vulnerabilities) | Precision/recall/F on real defects with a defined ground truth | Synthetic-only defects; unclear ground truth |
| Generation / synthesis (tests, code, patches) | Validity of the produced artifact (compiles, passes, holds the property) | Similarity-to-reference proxy instead of validity |
| Repair | Verified behavior change: re-run + oracle; assertion/spec preservation | "Plausible patch" without an overfitting check |
| Localization / ranking | Rank-based effectiveness on real faults vs. alternatives | Cherry-picked programs; one metric only |
| Scalability / performance | Real-system sizes, wall-clock with a fair config | Toy inputs; unequal baseline budget |
Real subject systems
- Use real software — open-source projects, real bug/defect datasets, real CI logs — not toy programs you constructed to make the tool look good.
- Report subject provenance: names, versions/commit SHAs, sizes, and the extraction date. Reviewers reproduce from this.
- Justify subject selection and disclose exclusions; self-selected subjects are the classic external-validity threat.
Fair, runnable tool baselines
- Compare against the closest runnable automation, configured at an equal, documented budget (time, iterations, tuning, seeds). ASE reviewers routinely rerun or scrutinize baselines.
- Pin baseline versions/commits and note reimplementation vs. original.
- If no tool baseline exists, construct a defensible non-trivial baseline (a static rewrite, a random or heuristic variant) rather than comparing only to "nothing."
Ablations that isolate the automation
If a learned or LLM component is involved, run an ablation that removes it and keeps the rest, so the marginal value of the design is visible. This is what defeats the "the model did it, not your technique" objection and keeps the paper ASE-shaped rather than ML-shaped.
Oracles and correctness
- State the oracle explicitly: how do you know a generated test is meaningful, or a repair is correct? Re-execution, differential testing, formal checks, or human audit — name it.
- For repair/synthesis, guard against overfitting to the evaluation oracle (e.g., patches that pass the given tests but break behavior): report a held-out or manual correctness check.
Statistics and effect sizes
- Report effect sizes and dispersion (confidence intervals, non-parametric tests where appropriate), not just point estimates or a single accuracy number.
- For randomized techniques (search-based, sampling, LLM temperature > 0), report repeated runs with variance and fix/seed the randomness for the artifact.
Contamination-aware LLM handling
- Record model identifiers and dates; a model updated between runs invalidates comparisons.
- Consider training-data contamination: benchmarks the model may have seen inflate results — report on held-out or post-cutoff subjects where feasible, and say so.
- Cache raw model outputs so the artifact reproduces rather than re-samples a live API.
Mining and dataset provenance
- Pin repository SHAs, the corpus extraction date, query/filter criteria, and any labeling protocol with inter-rater agreement for manually coded data.
- Version the dataset and describe how to regenerate it; a package that needs live scraping re-samples a moving target.
Evaluation audit checklist
[Claim-evidence] each claim -> a matching metric on real subjects (not a proxy)
[Subjects] real, provenance-pinned, selection justified, exclusions disclosed
[Baselines] closest runnable tool, version pinned, equal documented budget
[Ablation] learned/LLM component isolated; marginal value of the design shown
[Oracle] correctness defined; overfitting-to-oracle checked
[Stats] effect sizes + dispersion; repeated runs for randomized methods
[LLM] model IDs/dates recorded; contamination considered; outputs cached
[Repro] provenance pinned; dataset/tool versioned for the artifact
Output format
[Automation claim] detection / generation / repair / localization / scalability
[Evidence match] metric(s) that fit the claim, on real subjects
[Baseline fairness] closest tool, budget parity, versions
[Ablation + oracle] learned-component ablation present; correctness oracle stated
[Threats] subject selection / oracle validity / baseline fairness / contamination — bounded how?
Version History
- 9f86f09 Current 2026-07-19 14:19
