Golden test cases and scoring workflow

Prerequisites

• RaTeX output: scripts/update_golden_output.sh needs a Rust toolchain; PNG/SVG need KaTeX TTFs under fonts/ (script falls back to tools/lexer_compare/node_modules/katex/dist/fonts).
• KaTeX fixtures: run npm install in tools/golden_compare (Puppeteer + katex dist, including contrib/mhchem.min.js).
• Compare script: python3 tools/golden_compare/compare_golden.py needs pip install Pillow numpy.

1. Add test cases

• Main suite: edit tests/golden/test_cases.txt at repo root. One LaTeX formula per line; blank lines ignored; lines starting with # or % are comments.
• mhchem (\ce, \pu, …): edit tests/golden/test_case_ce.txt with the same rules.

Formula order defines case indices: 0001.png is the first non-comment formula, 0002.png the second, etc. (same ordering as in generate_reference.mjs and compare_golden.py).

2. Generate RaTeX output and KaTeX fixtures

From repo root:

./scripts/update_golden_output.sh

Builds ratex-render / render-svg, writes main-suite PNGs to tests/golden/output/ and SVGs to tests/golden/output_svg/; if test_case_ce.txt exists, also output_ce/ and output_svg_ce/ (mhchem uses --dpr 2 to match reference pixel density). It also writes a complete render-manifest.json in each PNG output directory, so failed cases remain present as indexed status records.

Generate KaTeX reference PNGs (fixtures):

cd tools/golden_compare
node generate_reference.mjs

Defaults: read tests/golden/test_cases.txt, write tests/golden/fixtures/. The generator is locked to KaTeX 0.16.45 and writes reference-manifest.json with actual KaTeX/Puppeteer/Chromium versions, DPR, font hashes, and one status record per formula.

mhchem suite:

cd tools/golden_compare
node generate_reference.mjs ../../tests/golden/test_case_ce.txt ../../tests/golden/fixtures_ce --mhchem

Note: generate_reference.mjs and update_golden_output.sh regenerate from the full case list. New cases are usually appended; rerun a full generation so indices stay aligned with NNNN.png filenames.

3. Compare scores and diff images

From repo root:

python3 tools/golden_compare/compare_golden.py

Defaults: tests/golden/fixtures/ vs tests/golden/output/ with tests/golden/test_cases.txt. The Python comparator is the only authoritative score source; Rust golden tests are smoke checks. It reports every formula index, including missing, failed, and policy-excluded cases.

mhchem:

python3 tools/golden_compare/compare_golden.py --ce

compare_golden.py arguments (reference)

Run from repo root so default paths resolve. All paths may be absolute or repo-relative.

Baseline comparison matches formulas by exact source text and duplicate occurrence order, so corpus additions, removals, and reordering do not turn otherwise comparable formulas into integrity errors.

Diff behavior: With --diff-dir only, diffs are written for failing cases (combined score strictly below --threshold). With --diff-from, diffs are written for every case in the index range (not only failures).

Copy-paste examples (repo root):

# Diffs only for failures (main suite)
python3 tools/golden_compare/compare_golden.py --diff-dir tests/golden/diffs

# Diffs for new cases 980–988 (1-based indices; adjust to your range)
python3 tools/golden_compare/compare_golden.py \
  --diff-dir tests/golden/diffs --diff-from 980 --diff-to 988

# Stricter pass bar + mhchem + failure diffs
python3 tools/golden_compare/compare_golden.py --ce --threshold 0.35 --diff-dir tests/golden/diffs_ce

Add tests/golden/diffs/ (or your chosen dir) to .gitignore unless the team commits diff PNGs for review.

For the compact main baseline, prefer ./scripts/update_golden_baseline.sh. It regenerates both image sides and writes the minified tests/golden/baseline.json. Generated output, manifests, CSV, and full diagnostic JSON are not committed; CI uploads the full report as an artifact.

Script arguments: update_golden_output.sh and generate_reference.mjs

scripts/update_golden_output.sh

• Paths are fixed inside the script (tests/golden/test_cases.txt, output/, output_svg/, and optionally test_case_ce.txt → output_ce/, output_svg_ce/).
• --png-only skips SVG builds/regeneration. The authoritative baseline helper and golden CI use this because the Python scorer consumes PNGs only.
• Requires repo root layout and font locations described in Prerequisites.

tools/golden_compare/generate_reference.mjs

Usage:

node generate_reference.mjs [test_cases.txt] [fixtures_dir] [--mhchem] [--manifest-out FILE]

mhchem (from tools/golden_compare):

node generate_reference.mjs ../../tests/golden/test_case_ce.txt ../../tests/golden/fixtures_ce --mhchem

Quick checklist

1. Edit test_cases.txt (or test_case_ce.txt).
2. ./scripts/update_golden_output.sh
3. cd tools/golden_compare && node generate_reference.mjs (mhchem: pass paths and --mhchem).
4. python3 tools/golden_compare/compare_golden.py --json-out … --fail-on-missing (mhchem: add --ce); use --diff-dir and optionally --diff-from / --diff-to for diff PNGs.