FSE Artifact Evaluation

Use this for the artifact track. FSE follows the ACM Artifact Review and Badging scheme, and the artifact evaluation is a separate, post-acceptance process with its own deadline. Two things to internalize: badges are earned by evaluators actually using your package, and the review artifact (anonymized, for the paper's reviewers) is not the same deliverable as the badge artifact (de-anonymized, permanently archived).

The ACM badges (verify the current set and names)

Available is a low-cost, high-value badge (archive the package); Functional/Reusable/Reproduced require the evaluator's own run to succeed, so the failure mode is always "did not run on their machine," never "the idea was weak."

What SIGSOFT evaluators open first

Assume an evaluator gives your package a bounded time budget on a clean machine. Design for the first ten minutes to succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
              "install these 40 things by hand"
[README]      one-screen orientation: what it is, how to install, how to run the demo, how to
              reproduce each claim, expected runtime and outputs
[Mapping]     an explicit table: paper claim -> script -> expected result
[Data]        the extracted dataset itself (or documented access), not just the query
[Provenance]  repo SHAs, extraction dates, model IDs/dates, seeds
[License]     an OSI-approved license so the artifact can be badged Reusable
[Archive]     deposit in a DOI-issuing repository for the Available badge

Anonymized review artifact vs. badge artifact

• At submission: the artifact is anonymized for the paper's reviewers — no owner strings, cluster paths, lab names, or identity-revealing links, and no live repository that discloses authors.
• After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version the artifact evaluators badge and the camera-ready cites.

Worked vignette: packaging a detection tool + study

A paper contributes a defect-detection tool and an empirical evaluation. To target Reusable and Reproduced: ship a Docker image with the tool pre-built; a run_demo.sh that detects on a small bundled project in under a minute; a reproduce/ directory whose scripts regenerate each table from logged results; a claim-to-script mapping table in the README; the extracted evaluation dataset with pinned SHAs; and an MIT/Apache license. State honestly which results are turnkey and which need the full (slow) dataset run.

Calibration

• The artifact deadline is after acceptance and independent of the camera-ready; do not conflate them.
• Badge names, the exact set offered, and whether evaluation is single- or double-anonymous vary by cycle — confirm on the current artifact-track call.

Output format

[Target badges] Available / Functional / Reusable / Reproduced
[Artifact role] anonymized review artifact / public badge artifact
[Contents] <tool/data/scripts/provenance/license>
[Ten-minute test] does install + demo succeed on a clean machine? yes/no
[Claim mapping] <claim -> script -> expected result present? yes/no>
[Fixes before upload] <ordered list>