Reproducing API bugs with property tests

tests/property/ fuzzes the /v3 API in-process against a real Postgres test database. Every test asserts that no response is a 5xx and that no unhandled exception escapes (TestClient re-raises them). Start here when an issue says "I got a 500 when…": the shortest path is usually to get one of these tests to fail on the reported input, then fix the cause.

The harnesses

They are skipped unless pytest gets --fuzz (CI passes it; the pre-push hook doesn't, to keep it fast), so every command below includes it. All of them use fuzz_client (one DB session per request, as in production) from tests/property/conftest.py.

Workflow

1. Try the broad fuzzer on the route first. Select by operation label and raise the example count:

   HONCHO_FUZZ_EXAMPLES=1000 uv run pytest tests/property/test_api_no_server_errors.py \
     --fuzz -k "messages and POST and not list and not upload" -n 0 -p no:cacheprovider \
     --tb=line --show-capture=no
   

   A failure prints the exception type and a curl command for the shrunk request.

2. If it passes, check that the route is actually being reached. A route that only ever gets 404/422 is only being fuzzed on its error paths:

   HONCHO_FUZZ_STATS=/tmp/stats.txt uv run pytest --fuzz tests/property/test_api_no_server_errors.py
   awk '{k=$1" "$2" "$3; t[k]++; if ($4<300) ok[k]++} END {for (k in t) if (!ok[k]) print k}' /tmp/stats.txt
   

   Fix coverage before concluding there is no bug: add the ID field to BODY_ID_FIELDS, seed the resource the route needs, or write a targeted strategy in test_api_inputs.py.

3. Pin the reported input. Put the issue's exact input in a test, either as an @example(...) on a property or as a plain test, and confirm it fails before you touch src/.

4. Fix the cause, not the status code. Follow the existing conventions:

   • NUL (\x00) is stripped at the schema boundary (NulStripped, strip_nul in src/utils/sanitization.py), not rejected. A new free-form text or JSON field needs the same treatment.
   • A handler that builds a create schema from a path ID (get-or-create of a parent workspace or session) goes through schema_from_path (src/utils/path_params.py), which turns a bad ID into a 422 at path.<param>. A ValidationError anywhere else is a server bug and stays a 500.
   • Values Postgres can't store are a 422 only for an explicit allowlist of SQLSTATEs (_is_input_data_error in src/main.py). Widen it only for an input class, and match narrowly when the SQLSTATE also covers server bugs (see the 22P02 surrogate case).
   • Library parse failures on uploads become a ValidationException, caught around the library call only.
   • Never add a blanket except Exception to turn a 500 into a 4xx.

5. Leave a deterministic regression test in tests/routes/ (most land in test_validation_api.py) that sends the shrunk input. Hypothesis runs with database=None, so a property failure is not replayed on the next run unless you pin it.

Gotchas

• Background work does not run under test (the deriver, deletion cleanup, the reconciler). Only the "deletion pending" state is reachable; LLM and embedding calls are mocked.
• DB state persists across Hypothesis examples within one test function. That helps (more collisions) but means a failure can depend on earlier examples; re-run with -n 0 and a fixed --hypothesis-seed to replay it.
• Schemathesis' positive mode respects the OpenAPI schema. When the server enforces a constraint the schema doesn't declare (e.g. peer_id must match the name pattern), positive-mode requests get 422s and never reach the handler. That gap is a schema bug in its own right.
• Default example counts are small so the suite stays fast. Raise HONCHO_FUZZ_EXAMPLES (and HONCHO_FUZZ_STEPS for the state machine) locally when hunting.