Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› eurosys-experiments
eurosys-experiments
GitHub指导EuroSys论文实验设计，构建四层证据栈，强调工作负载真实性、基线公平调优及可复现性矩阵规划。
触发场景
设计系统性能评估方案
审查论文实验部分的严谨性
选择具有代表性的工作负载
调整基线系统配置以确保公平对比
安装
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill eurosys-experiments -g -y
SKILL.md
Frontmatter
{
"name": "eurosys-experiments",
"description": "Use when designing or auditing the evaluation of a EuroSys paper — choosing workloads that earn the word realistic, tuning baselines beyond their defaults, decomposing end-to-end wins into per-mechanism gains, measuring overheads and worst cases, and sizing experiments to the claims the paper actually makes."
}
EuroSys Experiments
Use this while the evaluation is still designable. EuroSys reviewers treat the evaluation as the paper's testimony about itself: the design section says what should happen, and §Evaluation is cross-examination. Plan it as a set of questions with falsifiable answers, not as a benchmark tour.
The four-layer evidence stack
| Layer | Question it answers | Typical EuroSys instrument |
|---|---|---|
| End-to-end | Does the system win where users live? | Full application or serving workload, realistic scale |
| Decomposition | Which mechanism buys which fraction? | Factor analysis: enable components one at a time |
| Cost | What does the win spend? | Memory/CPU/network overhead, code and ops complexity |
| Boundary | Where does it stop winning? | Adversarial mixes, saturation, failure injection |
A submission with only the first layer reads as a demo; the middle layers are what convert "it is faster" into "we understand why it is faster", which is the systems-research standard of proof.
Workload realism, argued not asserted
- Prefer published traces and standard suites with named versions; when using a synthetic generator, calibrate it against a real distribution and say how.
- Match the workload to the claim's regime: a memory-efficiency claim needs memory pressure; a tail-latency claim needs load near saturation, not at 30%.
- Scale honestly: if the pitch is rack-scale, a two-node result needs an explicit extrapolation argument or a scoped-down claim.
- Report the workload's own parameters (skew, read ratio, arrival process) so the experiment is reconstructible without your cluster.
Baseline fairness — the venue's sharpest knife
The reflexive EuroSys reviewer question is "did they tune the baseline?" Answer it before it is asked:
- Run baselines at their documented best configuration for your hardware, and cite where that configuration comes from.
- Include the strongest deployed alternative, not only research prototypes — losing to a well-tuned production system on some axis is survivable; omitting it is not.
- Same hardware, same workload generator, same measurement harness for every system; any asymmetry gets a sentence of justification.
- When your system loses a metric, plot it anyway and explain the tradeoff.
Experiment matrix as an artifact
Freeze the plan in a machine-checkable form before running:
# eval-matrix.yaml — one row per claim the paper will make
- claim: "cuts p99 GET latency ≥40% under skewed load"
figure: fig8
workload: {trace: twitter-cache-2020, skew: zipf-0.99, load: 0.85sat}
systems: [ours@v1.4, baselineA@3.2-tuned, baselineB@1.9-tuned]
reps: 10
metrics: [p50, p99, p999, throughput]
status: pending
The matrix doubles as the reproducibility ledger (eurosys-reproducibility)
and later as the artifact's claims map (eurosys-artifact-evaluation).
Result invalidators to design out early
Each of these has sunk otherwise strong EuroSys evaluations; each is cheap to prevent and expensive to discover in a review:
- Measuring through a bottleneck that is not the system under test — a saturated client, a slow log disk, a debug build of a dependency.
- Comparing your warm system against cold baselines (or vice versa) because warm-up policy was never standardized across systems.
- A "scalability" curve whose x-axis grows load and resources together, so nothing about scaling behavior is actually isolated.
- Reporting means over runs that include documented failures — decide the failure-handling policy for measurements before running them.
- Config drift between the end-to-end and decomposition experiments, so the component gains do not sum to anything resembling the headline.
Sizing runs for the claim
- Comparative bar charts: ≥5–10 repetitions with dispersion shown; single-run bars invite a one-line rejection rationale.
- Tail latencies: enough requests that the quoted percentile has support — a p999 from 10k requests is ten samples of noise.
- Ablations: vary one factor per experiment; a config delta of two changes attributes nothing.
- Timeboxing: reserve cluster time for the boundary layer explicitly; it is always the layer teams drop under deadline pressure and the layer reviewers miss loudest.
Reporting floor for the paper
- Hardware and topology for every experiment, once, in a table the reader can find (§Evaluation setup), with per-figure deviations noted.
- Software versions for every system in every comparison, including yours.
- Load points quoted with the metric ("85% of measured peak throughput"), never as bare client counts whose meaning depends on the cluster.
- Repetition count and dispersion type in every caption that shows a comparison.
- A sentence on measurement methodology for anything subtle: how latency is timestamped, where the measurement harness sits, what it costs.
Output format
[Claim -> evidence map] <each paper claim: figure, workload, systems, reps>
[Stack coverage] end-to-end / decomposition / cost / boundary: present or missing
[Baseline fairness] <tuning provenance and deployed-alternative status>
[Realism audit] <trace provenance, load regime, scale honesty>
[Highest-value missing run] <the one experiment to schedule next>
版本历史
- 9f86f09 当前 2026-07-19 15:41
