Agent Skills
› brycewang-stanford/Awesome-Journal-Skills
› vis-experiments
vis-experiments
GitHub用于IEEE VIS论文提交前的实验评估审计，指导根据贡献类型匹配证据标准。涵盖控制实验设计、功效分析、CVD安全编码选择、公平基准测试及定性研究规范，确保证据与主张一致并满足审稿要求。
触发场景
设计或审计IEEE VIS论文的实验部分
需要验证评估证据是否匹配特定贡献类型
进行受控实验设计或功效分析
选择感知安全的视觉编码方案
安装
npx skills add brycewang-stanford/Awesome-Journal-Skills --skill vis-experiments -g -y
SKILL.md
Frontmatter
{
"name": "vis-experiments",
"description": "Use when designing or auditing IEEE VIS evaluations, covering how to match evidence to the contribution type (perceptual study, controlled user study, algorithm benchmark, design-study validation, qualitative work), controlled experiment design with power and effect sizes, CVD-safe and perceptually grounded encoding choices, task taxonomies, and provenance so a TVCG reviewer trusts the result."
}
VIS Experiments
Use this before submission when the evaluation is not yet locked. IEEE VIS reviewers judge whether the evidence matches the contribution type — and visualization has several distinct contribution types, each with its own evidence standard. The organizing principle is evaluate the claim you actually make: a claim about perception needs a controlled study, a claim about scale needs a benchmark, a claim about real-world usefulness needs a design-study validation or a deployment.
Evaluation audit
- Pick the evaluation to the contribution type, not by habit (see the table). The classic VIS reject is a system paper "evaluated" only by an accuracy number, or a perceptual claim backed only by author intuition.
- Design controlled studies properly: state hypotheses, a within/between design, a task from a
recognized task taxonomy, a power analysis justifying N, and report effect sizes with
confidence intervals, not just p-values. Consider preregistration for confirmatory studies
(
vis-reproducibility). - Justify encodings perceptually: color choices should be CVD-safe and appropriate to the data type (sequential/diverging/categorical); channel choices should follow known effectiveness rankings for the task. Reviewers check this explicitly.
- Benchmark techniques fairly: compare against the strongest existing technique and a reasonable baseline on realistic data sizes, with runtime/quality reported and the code available.
- Hold qualitative and design-study work to method: coding schemes, multiple coders, reflection across abstraction levels, and an audit trail — design studies are a first-class VIS contribution, not a weak substitute for a controlled study.
- Pin provenance for datasets, stimuli, and rendering so the evaluation reproduces rather than re-samples.
Contribution-type to evidence table
| Contribution type | Matching evidence | Reject pattern avoided |
|---|---|---|
| Perceptual/cognitive claim | Controlled experiment: real stimuli, power analysis, effect sizes + CIs | "Author intuition stands in for a perception result" |
| New encoding/interaction technique | Controlled study and/or task-based comparison vs. the conventional design | "Prettier, but no evidence it helps a task" |
| System / tool | Demonstration of real use, expert feedback, or a usage study | "Feature list with no evaluation of use" |
| Design study | Reflection + validation across data/task/encoding/algorithm levels | "A one-off tool with no transferable lesson" |
| Algorithm (layout/rendering) | Benchmark: quality + runtime vs. strong baselines on realistic sizes | "Toy inputs only; no comparison" |
| Data/model contribution | Characterization + a task the data enables, with the data shared | "Dataset dumped with no analysis or task" |
Controlled-study design floor
[Hypotheses] stated before analysis; confirmatory vs. exploratory labeled
[Design] within/between justified; counterbalancing; the task from a known taxonomy
[Power] an a-priori power analysis justifies N; do not stop at "we recruited 20"
[Stimuli] real or realistic; the exact stimuli archived
[Measures] accuracy AND time AND (where relevant) preference/confidence; define each
[Statistics] effect sizes + CIs; appropriate tests; corrections for multiple comparisons
[Reporting] report what you found, including null and exploratory results, honestly
Perceptual and accessibility checks
- Color: use CVD-safe palettes; match palette type to data (sequential for ordered, diverging for a meaningful midpoint, categorical for nominal); never encode magnitude on hue alone.
- Channel effectiveness: prefer position/length for quantitative comparison; justify any use of area, angle, or color for a precise task.
- Legibility: ensure figures read in grayscale and at print size; a result a reviewer cannot see is a result you cannot claim.
Vignette: evaluating a new time-series encoding
Suppose the paper claims a new encoding reads trends faster than a line chart. The matching plan: a controlled within-subjects study; trend-reading tasks drawn from a task taxonomy; real time-series stimuli, archived; an a-priori power analysis fixing N; accuracy and completion-time as measures; effect sizes with CIs comparing the new encoding to a tuned line-chart baseline; a CVD-safe palette justified against the task; and honest reporting of any task where the line chart won — every number traceable to the archived analysis notebook.
Output format
[Evaluation readiness] strong / adequate / weak
[Contribution type] perceptual / technique / system / design-study / algorithm / data
[Evidence match] <contribution type -> evidence chosen -> appropriate? yes/no>
[Study rigor] <hypotheses? power analysis? effect sizes + CIs? preregistered?>
[Encoding validity] <CVD-safe? channel matched to task? grayscale-legible?>
[Provenance] <stimuli/data/rendering archived and reproducible? yes/no>
[Decision-critical next run] <one study or benchmark to add>
版本历史
- 9f86f09 当前 2026-07-19 17:53
