test-napkin-math
GitHub用于验证 napkin_math 实验相关代码变更的冒烟测试工具，覆盖蒙特卡洛运行、确定性、算术逻辑及 Schema 校验，确保改动未破坏核心功能。
Trigger Scenarios
Install
npx skills add PlanExeOrg/PlanExe --skill test-napkin-math -g -y
SKILL.md
Frontmatter
{
"name": "test-napkin-math",
"description": "Use after any change under experiments\/napkin_math\/ or to the upstream skill prompts that feed into it (extract-parameters-from-full, extract-parameters-from-digest, generate-bounds, generate-calculations, run-scenarios, monte-carlo). Runs the smoke-test suite and reports pass\/fail. Invoke before declaring napkin-math work done."
}
Test the napkin_math experiment
Overview
A single-shot smoke check for the napkin_math experiment. The test logic lives in experiments/napkin_math/tests/run_smoke.py; this skill is a thin wrapper that invokes the script, parses its output, and reports a one-line summary.
Covers the Monte Carlo runner (end-to-end, determinism, Bernoulli arithmetic, sensitivity ranking), the strict-schema fail-fast paths (each required field individually), prepare_extract_input.py import sanity, and the compress_report_section pytest suite.
When to Use
- After any edit to:
experiments/napkin_math/run_monte_carlo.pyexperiments/napkin_math/prepare_extract_input.pyexperiments/napkin_math/tests/fixtures/smoke/*worker_plan/worker_plan_internal/parameter_extraction/compress_report_section.py- any system-prompt.txt / SKILL.md under
experiments/napkin_math/.claude/skills/{extract-parameters-from-full,extract-parameters-from-digest,generate-bounds,generate-calculations,run-scenarios,monte-carlo}/that touches the artifact schema
- Before declaring any napkin_math change "done" — even if the change looks self-contained, the schema is tightly coupled across stages
- When the user asks "did I break anything?" or "run the napkin_math tests"
Not for: end-to-end LLM-driven runs of the extract-parameters-from-full skills against real reports (those require an LLM in the loop and are out of scope here).
Workflow
-
Invoke the runner. Requires Python 3.11+ with NumPy and pytest installed:
/opt/homebrew/bin/python3.11 experiments/napkin_math/tests/run_smoke.pyOverride the interpreter with
NAPKIN_TEST_PYTHON=<path>if the default isn't available. -
Read the script's stdout. It prints one section per check with individual
ok/FAILlines, then a final summary:SUMMARY: 7/7 checks passed ALL GREENExit code is
0on full pass,1on any failure. -
Report back to the user. On success: one line ("All 7 napkin_math smoke checks passed."). On failure: list the failing checks with the detail printed by the runner, and point at the specific file or schema field implicated.
What the seven checks cover
| Check | What it verifies |
|---|---|
end_to_end |
Runner consumes the synthetic fixture, emits both expected outputs, zero warnings |
determinism |
Two runs with the same seed produce byte-identical JSON |
bernoulli_arithmetic |
Mean of total_budget_with_gate_inr ≈ 1,000,000 + 0.6 * 250,000 within ±5,000 |
sensitivity_ranking |
Bernoulli gate is the sole driver of its dependent output (correlation == 1.0); two-input convert sensitivity has both inputs |
schema_errors |
Each required schema field (sampling_discipline, non_negative, default_pass_probability for bernoulli_gate, output_name, output_unit) triggers a SCHEMA ERROR exit code 2 with a message naming the upstream stage to re-run |
prepare_extract_input_imports |
prepare_extract_input.py imports cleanly and exposes build_combined_digest |
compress_pytest |
The pytest suite for compress_report_section.py passes (13 tests) |
Common Mistakes
| Mistake | Fix |
|---|---|
| Skipping the check because "the diff is tiny" | The schema is tightly coupled across five skills + the runner. A tiny change in one prompt can break the runner's strict validation. Always run it. |
| Reporting "tests pass" without running the script | If you didn't see ALL GREEN, you didn't pass. |
Running with python3 instead of python3.11 |
NumPy and the worker_plan tests live in the 3.11 env. Use the explicit path or set NAPKIN_TEST_PYTHON. |
Treating a compress_pytest failure as unrelated |
The compressor is part of the napkin_math experiment; if its tests break, the digest produced by prepare_extract_input.py is suspect. |
| Editing fixtures to make tests pass | Fixtures encode the contract. If a fixture needs to change, the schema or the runner changed for a real reason — update both consciously and re-justify each touched check. |
Reference
- Test runner:
experiments/napkin_math/tests/run_smoke.py - Synthetic fixture:
experiments/napkin_math/tests/fixtures/smoke/ - Monte Carlo runner under test:
experiments/napkin_math/run_monte_carlo.py - Companion skills (consumers of the same schema):
../extract-parameters-from-full/SKILL.md,../extract-parameters-from-digest/SKILL.md,../generate-bounds/SKILL.md,../generate-calculations/SKILL.md,../run-scenarios/SKILL.md,../monte-carlo/SKILL.md
Version History
- 846f612 Current 2026-08-20 11:18
