webconf-experiments
GitHub指导WWW会议论文实证设计，确保证据与主张匹配。涵盖数据集溯源、新鲜度要求及统计报告；重点审计时间、流行度和数据污染泄露；规范基线选择与消融实验，杜绝方法学缺陷。
Trigger Scenarios
Install
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill webconf-experiments -g -y
SKILL.md
Frontmatter
{
"name": "webconf-experiments",
"description": "Use when designing or auditing the empirical section of a Web Conference (WWW) paper — matching evidence to the claim's scale, choosing datasets with provenance and freshness, blocking temporal and popularity leakage, running honest baselines from the sibling circuit, and deciding when live-platform or user-study evidence is required."
}
Web Conference Experiments
The empirical bar at this venue is not "more datasets" — it is evidence whose scale, freshness, and realism match the claim. A method claiming web-scale efficiency must show the scaling curve; a measurement claiming platform-general behavior must show more than one platform; a system claiming deployability must show cost under realistic load. Audit the claim-evidence match before adding anything.
Claim-to-evidence contract
| Claim type | Minimum honest evidence | Habitual shortfall |
|---|---|---|
| "Outperforms" (quality) | Tuned recent baselines, repeated runs, variance, significance | Untuned baselines from 3-year-old code |
| "Scales" (efficiency) | Cost curves across ≥2 orders of magnitude, hardware stated | One big-dataset wall-clock number |
| "Generalizes" (external validity) | ≥2 platforms/domains or a stated single-platform scope | Silent single-platform universality |
| "Measures" (phenomenon) | Construct definition, sampling frame, bot/spam handling, error bars | Convenience crawl treated as census |
| "Deploys" (system) | Load, latency percentiles, failure behavior; A/B where claimed | Demo-grade throughput on toy traffic |
Dataset selection with provenance
- Prefer datasets with documented collection: source, crawl window, sampling rule, known biases. An undocumented Kaggle mirror of a platform dump is a provenance liability reviewers now flag.
- Freshness is evidential here: user behavior and platform mechanics drift, so a 2015 interaction dataset supports historical claims, not claims about current platforms. Either add a recent corpus or scope the claim in time.
- Report per-dataset statistics (nodes/edges/users/items/time span) in a table, and state why this set of datasets spans the claim — heterogeneity in domain, size, and density is the point, not the count.
- For data you collected: consent/ToS posture and aggregation level belong beside
the dataset description (body text, per
webconf-writing-style).
Leakage: the venue's most-caught methodological bug
Web data is temporal, popularity-skewed, and duplicated across the crawl. Three specific leaks to audit:
- Temporal: random splits on interactions or evolving graphs train on the future. Use a time-based split rule and state it ("train < T ≤ test").
- Popularity: negative sampling that mirrors test popularity inflates ranking metrics; report at least one popularity-debiased or full-ranking metric.
- Duplication/contamination: near-duplicate pages across splits, and — for LLM-era pipelines — benchmark text present in a foundation model's pretraining. State the dedup rule; where an external LLM is a component, discuss contamination explicitly rather than hoping no one asks.
Baselines and ablations
Baselines come from the sibling circuit's current cycle (WWW, WSDM, SIGIR, KDD, CIKM within ~2 years), tuned with the same budget as your method — state the search space for both, since asymmetric tuning is the most common quiet unfairness. Ablate the claimed mechanism, not everything: if the paper's story is "the crawl-time signal is what helps," the ablation table must contain exactly the row that removes it.
Experiment matrix skeleton (fill per claim, not per dataset):
claim C1 "quality" : datasets D1-D3 x {ours, B1..B4} x 5 seeds -> mean±sd, test
claim C2 "scales" : D_synthetic sized 10^6..10^9 edges x {ours, B1} -> cost curve
claim C3 "mechanism": ours minus {signal S, module M} on D1-D3 -> delta table
claim C4 "external" : platform P2 replication of headline result -> one table
Every table row answers a named claim; rows answering nothing get cut.
Statistics that survive review
- Repeated runs with different seeds wherever training is stochastic; report mean ± sd and the number of runs.
- Significance tests matched to the design (paired across datasets/queries), with effect sizes — at web scale everything is "significant," so the delta's magnitude and cost carry the argument.
- Percentiles, not means, for latency and exposure-type quantities; web distributions are heavy-tailed and means mislead.
When offline evidence is not enough
Claims about user response (satisfaction, engagement shifts, behavior change) need a user study or an online experiment, with ethics posture stated; claims about production viability need load realism. If neither is obtainable, weaken the claim to what offline evidence supports — "improves offline ranking quality" is publishable; "improves user satisfaction" without users is a rebuttal-week wound that cannot be healed in seven days.
Predictable objections and their cheap preemptions
Four objections recur across this venue's tracks; each has a preemption that costs a sentence or a table row, not a new experiment campaign:
- "Gains may come from capacity, not the proposed component" — include one parameter-matched ablation row and state parameter counts in the caption.
- "Datasets are all from one platform family" — either add the second-family corpus or write the scope sentence in the conclusion; silence converts a limitation into a discovered flaw.
- "The baseline numbers differ from the original paper" — declare the re-implementation and the protocol difference (split, metric variant, preprocessing) in a footnote at first mention; unexplained deltas read as either sloppiness or gaming.
- "Efficiency claims lack a cost axis" — every quality table involving a method whose selling point includes scale gets a companion column (time, memory, or queries), because at web scale a 0.5-point win at 10x cost is a negative result.
The meta-rule: reviewers here are drawn from a circuit that reviews the same methods at WSDM, SIGIR, and KDD in the same year; they have seen this quarter's common failure modes several times already. The paper that names its own weaknesses first is the one whose rebuttal week is quiet.
Audit checklist
- Every headline claim mapped to a table/figure that can carry it.
- Split rules stated; temporal/popularity/duplication leaks addressed.
- Baselines recent, tuned symmetrically, search spaces disclosed.
- Variance, tests, and effect sizes present; percentiles for tails.
- Scope sentences where evidence is single-platform or offline-only.
Output format
[Claim-evidence map] C1..Cn -> table/figure or GAP
[Leakage audit] temporal / popularity / duplication: clean or findings
[Baseline fairness] recency, tuning symmetry: pass/fail
[Scale realism] does evidence scale match claim scale? <notes>
[Required additions] <ranked by review risk, with cost estimate>
Version History
- 9f86f09 Current 2026-07-19 17:50
