/visual-data-dictionary

Scope: repo-local. This lives beside build-dashboard, release-prep and review-respond at the top of .claude/skills/, which package-plugin.js and package-mcpb.js do not archive — they ship only .claude/skills/skills/. So this skill is available when working in the qsv repo and is not part of the distributed plugin. That is deliberate: the packaged skills drive qsv through the mcp__qsv__* MCP tools, while this one drives the qsv CLI directly and needs Bash plus python3. Shipping it would require rewriting it against the MCP tool surface, which has no equivalent for the GeoJSON inspection or the HTML verification below (nor for the optional Stage 6 browser pass).

Requires: qsv on PATH, python3, and an LLM endpoint for describegpt. Stage 6 (optional) additionally needs a browser-automation MCP — any one (Playwright MCP, claude-in-chrome, …); skip the stage when none is available.

Turn a CSV into a self-contained HTML Data Schematic whose panels are chosen from an LLM-inferred data dictionary, with that dictionary embedded beside the charts.

Four stages, plus one optional fine-tune and one optional browser pass, in this order and no other:

1. denull — blank null sentinels so numeric columns are actually numeric
2. describegpt — infer a JSON Schema data dictionary from the cleaned data
   • 2.5 fine-tune (optional) — hand-correct the dictionary in a terminal UI before it drives the Data Schematic
3. geojson (optional) — fetch US boundaries with --geojson auto, or pick a feature id key by inspecting a supplied file
4. viz smart — render the Data Schematic, dictionary-driven, dictionary-embedded
   • 5 verify — check the HTML, then report
   • 6 tour refinement (optional, browser) — step through the guided Tour and refine its x-qsv.tour narration against what actually rendered

The order is load-bearing. Clean first, then describe, then draw. A dictionary built from dirty data documents a String column that is really a number, and viz smart will then chart it as a category or skip it outright.

IMPORTANT

You must execute bash commands. Never invent qsv flags — if unsure, run qsv <cmd> --help. Skip any step already satisfied by conversation context. Defer to CLAUDE.md when it conflicts with this skill.

Naming

Given input data.csv, derive:

Never write to the input path. denull --apply refuses to overwrite its own input (it compares file identity, so a hard link is caught too), but pick a distinct -o anyway.

Stage 0 — Preconditions

command -v qsv >/dev/null || { echo "qsv not on PATH"; exit 1; }
test -f "$INPUT" || { echo "no such file: $INPUT"; exit 1; }
qsv headers "$INPUT" | head -30
qsv count "$INPUT"

Only CSV/TSV/SSV. If handed a spreadsheet, convert first (qsv excel).

Build the index and stats cache once — every later stage reuses them:

qsv index "$INPUT"
qsv stats "$INPUT" --everything --stats-jsonl --force > /dev/null

Stage 1 — denull

Report first. Always show the user before changing their data.

qsv denull "$INPUT"

Read the verdict column:

• No rows, or no confirmed row → nothing to clean. Set WORK="$INPUT" and go to Stage 2. Do not create a copy.
• One or more confirmed → show the table, then:

qsv denull --apply "$INPUT" -o "${STEM}.denulled.csv"
qsv index "${STEM}.denulled.csv"
qsv stats "${STEM}.denulled.csv" --everything --stats-jsonl --force > /dev/null

Set WORK="${STEM}.denulled.csv".

--apply prints its report to stderr and the cleaned CSV to -o, and blanks sentinels only in the columns it confirmed. Every other column is copied through byte-for-byte.

Sanity check worth doing: each confirmed column's rows_affected should equal its nullcount in the new stats.

qsv stats "${STEM}.denulled.csv" | qsv select field,type,nullcount | qsv table

Two things to tell the user, because they are not obvious:

• denull only confirms columns that would promote to a numeric type once blanked. A categorical column holding NULL (e.g. status = ok/pending/NULL) is deliberately left alone — blanking it promotes nothing. Stage 2 will still surface it.
• Numeric sentinels (-999, 9999) are not detectable by any scan: they parse as valid numbers. Only Stage 2's LLM can propose them, and only a human should apply them.

Stage 2 — describegpt → JSON Schema dictionary

Resolve the LLM endpoint

Detect, then prompt only if nothing is found. Do not print key values.

for v in QSV_LLM_BASE_URL OPENAI_API_KEY QSV_LLM_APIKEY ANTHROPIC_API_KEY; do
  val=$(printenv "$v" 2>/dev/null); [ -n "$val" ] && echo "$v is set"
done
curl -s -m 2 http://localhost:1234/v1/models >/dev/null 2>&1 && echo "LM Studio on :1234"
curl -s -m 2 http://localhost:11434/api/tags  >/dev/null 2>&1 && echo "ollama on :11434"

Both LM Studio and ollama speak the OpenAI-compatible API, so both list models the same way and both take a /v1 base URL. Only the port differs:

# honor an explicit QSV_LLM_BASE_URL first; only probe local servers when it is unset
BASE_URL="${QSV_LLM_BASE_URL:-}"
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:1234/v1/models  >/dev/null 2>&1 && BASE_URL=http://localhost:1234/v1
[ -z "$BASE_URL" ] && curl -s -m 2 http://localhost:11434/api/tags >/dev/null 2>&1 && BASE_URL=http://localhost:11434/v1

[ -n "$BASE_URL" ] && curl -s "$BASE_URL/models" \
  | python3 -c 'import sys,json;[print(m["id"]) for m in json.load(sys.stdin)["data"]]'

/api/tags is only a liveness probe for ollama — it returns ollama's native shape, not the OpenAI {"data":[...]} envelope. List models from /v1/models either way.

If nothing is found, use AskUserQuestion for base URL + model. Never guess a model name. Offer the models the server actually reports; do not type one from memory.

Generate

First ask with AskUserQuestion: "Who is the Data Schematic's guided Tour for?" — default TOUR_AUDIENCE="Explain like I'm 10"; any free-text audience works ("a board of directors", "data journalists", …).

qsv describegpt "$WORK" \
  --dictionary --description --two-pass --infer-content-type \
  --format JSONSchema \
  --tour-audience "$TOUR_AUDIENCE" \
  ${BASE_URL:+--base-url "$BASE_URL"} --model "$MODEL" \
  -o "$SCHEMA"

• --infer-content-type is mandatory here, not optional: viz smart routes panels off each field's role and concept, and those are only inferred under this flag. Without it the dictionary loads and changes nothing. It is also the only way to get the three dictionary hints that unlock extra panels: per-field x-qsv.gauge_range (turns a measure's KPI tile into a gauge; kept only when the observed data lies inside the range), per-field x-qsv.denominator on a region column (adds a per-capita rate map beside the raw count map), and the dataset-level x-qsv.relationships array, whose "kind": "pipeline" entry is the only source of the pipeline funnel/bridge panel.
• x-qsv.denominator is the odd one out: qsv derives it, the LLM does not propose it. The model's whole contribution is tagging one column measure.population (a count of people or households IN a region another column names); describegpt then attaches the hint to every region column that can hold it, checking from the stats cache alone that the counts are plausible. Two population-shaped columns are an ambiguity it refuses rather than guesses at, so you get nothing. This is what makes a rate map reachable without hand-editing the JSON or paying for a --denominator census fetch of a number the file already carries.
• Pass --context-file <file> when the user has a glossary, README or codebook. Better context yields better roles, concepts and labels, hence a better Data Schematic. (viz --dictionary-context is the same thing for the infer path, which this skill does not take.)
• --two-pass roughly doubles cost and latency. It is what lets the model relate fields to one another (street_no + street + city + zip = one address), which is what makes the routing good.
• Naming it <WORK stem>.schema.json means a later qsv viz smart "$WORK" --dictionary infer finds and reuses it instead of paying for the LLM again. Delete the file to force a re-infer.
• --tour-audience makes describegpt also write a dataset-level x-qsv.tour narration — the prose the Data Schematic's guided Tour speaks — in the audience's register. The audience shapes ONLY the tour prose; labels and descriptions keep their normal register. Stage 6 refines it in a browser.

Optionally add --infer-null-values to have the model propose null sentinels into each property's x-qsv object, split into null_values (confirmed present by qsv) and null_candidates (guesses, each stamped confirm_required: true). This is the only route to numeric sentinels like -999. It is reported, never applied — nothing downstream acts on it.

Verify the dictionary carries what viz needs before spending time on Stage 4:

python3 - "$SCHEMA" <<'PY'
import json, sys
s = json.load(open(sys.argv[1]))
p = s["properties"]
have = sum(1 for v in p.values() if v.get("x-qsv", {}).get("role"))
print(f"role/concept on {have}/{len(p)} columns")
if have == 0:
    print("WARNING: no roles inferred — was --infer-content-type passed?")
# panel-unlocking hints, so the user knows up front what will/won't be drawn
gauges = [k for k, v in p.items() if (v.get("x-qsv") or {}).get("gauge_range")]
# a derived denominator turns the region map into a count map PLUS a rate map
denoms = {k: ((v.get("x-qsv") or {}).get("denominator") or {}).get("column")
          for k, v in p.items()}
denoms = {k: c for k, c in denoms.items() if c}
# viz reads pipelines ONLY from the dataset-level x-qsv (see xq_pipelines in
# src/cmd/viz.rs) — a root-level "relationships" array draws nothing.
rels = (s.get("x-qsv") or {}).get("relationships") or []
pipes = [r for r in rels if r.get("kind") == "pipeline"]
print(f"gauge_range on {len(gauges)} measure(s): {', '.join(gauges) or '(none)'}")
print(f"denominator on {len(denoms)} region column(s): "
      + (", ".join(f"{k} -> {c}" for k, c in denoms.items()) or "(none)")
      + ("   [rate map]" if denoms else ""))
print(f"relationships: {len(rels)} ({len(pipes)} pipeline -> funnel/bridge panel)")
if not rels and s.get("relationships"):
    print("WARNING: relationships found at the ROOT, not under x-qsv — viz ignores"
          " those. Is this the flat JSON dictionary instead of JSONSchema?")
tour = (s.get("x-qsv") or {}).get("tour")
if tour:
    print(f"tour: version {tour.get('version')}, audience {tour.get('audience')!r}, "
          f"{len(tour.get('overrides') or {})} override(s), "
          f"{len(tour.get('panels') or {})} panel narration(s)")
else:
    print("tour: (none — was --tour-audience passed?)")
PY

No gauge_range, no denominator and no pipeline is a perfectly normal outcome — most datasets have none of a canonical-scale measure, a per-region population, or a staged process. Say so and move on; all three can be hand-added later (see Stage 2.5).

Stage 2.5 — Fine-tune the dictionary (optional, TUI)

describegpt is a good first draft, not gospel — and the draft is not even stable: because the semantic half comes from an LLM, inferring twice over the same data can return different role/concept assignments, and role decides which panel a column gets (qsv issue #4407). This stage is what makes a Data Schematic reproducible. The corrected dictionary — reviewed, kept beside the data, committed if the data is versioned — is the artifact of record; every later run reuses it instead of re-rolling the model.

The five fields that actually steer viz smart — x-qsv.role, x-qsv.concept, title (label), description and x-qsv.aggregation — are worth a human pass when the model mislabels a column: a code that should be an identifier charted as a measure, a geo.* key left unknown, a per-unit price summed into a meaningless total, a bland label. edit_dictionary.py (beside this SKILL.md) is a curses UI that walks every column and, as you edit, previews how viz smart will route it (Skip / Dimension / Temporal / MapCoord / ProjectedCoord / Measure — the last showing its aggregation, Measure(sum) for an additive amount, Measure(mean) for a ratio, a duration, or anything you tag aggregation: mean), so you see the effect before rendering. It touches only those five fields, preserves every other key, and rewrites the file only if you save.

aggregation is the one field qsv can also drop on read, and the ! flag mirrors exactly when that happens: the token must be sum/mean, x-qsv.qsv_type must be Integer/Float (or absent), x-qsv.role must be empty or exactly measure, and the column must route to a measure at all. A value failing any of those is flagged and left out of the ROUTE preview, because viz silently falls back to its own name heuristic there. It catches the three easy mistakes: "average" instead of "mean", an aggregation left behind on a column you just re-roled to dimension, and one on a column nothing classifies (Defer→stats), where viz's stats floor discards it outright.

Offer it with AskUserQuestion: "Hand-tune the data dictionary in a TUI before rendering?" If no, go straight to Stage 3 — but say plainly that the Data Schematic then rests on an unreviewed draft, and that the dictionary can be tuned and re-rendered at any time without paying for the LLM again.

If yes, you cannot drive it yourself — a curses TUI needs the user's real terminal, and your Bash tool is a captured, non-interactive shell (the script detects this and refuses). So run it out-of-band:

1. Show the current routing so the user knows the starting point (this works without a TTY):

   python3 "$SKILL_DIR/edit_dictionary.py" --summary "$SCHEMA"
   

   where $SKILL_DIR is this skill's own directory (the folder holding this SKILL.md).

2. Tell the user to run this in their own terminal, then end your turn and wait — do not proceed:

   python3 "<skill dir>/edit_dictionary.py" "<SCHEMA path>"
   

   Keys: ↑↓ move · r role · c concept · l label · d description · a aggregation · s save · q quit. role/concept open a filterable picker (type to filter; off-vocab values are allowed but flagged with *). aggregation offers sum / mean / clear only — that is the whole vocabulary qsv accepts — and clearing removes the key rather than blanking it, restoring qsv's own guess.

3. When the user says they're done, re-read the file: re-run the Stage 2 coverage check and the --summary above, and show a short before/after of any rows whose role/concept/route changed. Then continue to Stage 3.

Because the dictionary keeps its <WORK stem>.schema.json name, Stage 4 picks up the edited file with no extra wiring. If the user edits nothing, the file is untouched byte-for-byte — treat that as a normal "looks good" outcome.

Scope note: the TUI deliberately does not edit null sentinels (--infer-null-values output). Those are reported-never-applied and have no viz smart effect, so editing them here would change nothing downstream.

Seven keys that do affect the Data Schematic are outside the TUI, and are hand-edited in the JSON — this is the supported path for them, not a violation of the "never hand-write the schema" rule. (x-qsv.denominator is the one qsv now normally fills in for you; hand-editing it is a correction, or the route to an area/household denominator describegpt will not derive.)

(x-qsv.aggregation used to be a fifth row here; it is now edited with a in the TUI above. Its meaning is unchanged: sum or mean on a numeric measure, declaring how the column combines across a group and overriding qsv's column-NAME heuristic in both directions. Use mean for anything per-unit or per-record — a unit price, a rating, a temperature, a duration — and sum only for a quantity each row contributes. It is the language-neutral signal: the name heuristic is English-first and cannot read non-English column names, issue #4401.)

One exception to "overrides in both directions" (issue #4528): a REGION-LEVEL column — one a region declares as its x-qsv.denominator, or one tagged measure.population/measure.area — holds a value that repeats identically on every row of its region. viz smart collapses it to one value per owning region before aggregating, and that collapse outranks an explicit x-qsv.aggregation. #4401's precedence is about what a measure MEANS (extensive vs intensive); this is about the data's SHAPE — the rows are duplicates of one regional value, and even a genuinely extensive measure must not be counted once per duplicate. So writing aggregation: sum on a population column does not restore a row-wise total. On tidy one-row-per-region data the collapse is a no-op. Two consequences to expect: its grouped bar shows the region's own figure (or, grouped by something coarser, the sum of the distinct regional values), and its KPI tile is omitted entirely when regions repeat, because no available aggregation can state a true dataset-wide total without a region-keyed dedupe pass.

For a pipeline, both encodings are hand-editable — stages as columns ("members" in process order, widest/upstream first, the opposite direction from "kind":"ordered"), or stages as row values ("stage_column" + an ordered "stages" list + an optional "value_column" to sum). Declared order is authoritative: if a stage outruns its predecessor, viz draws a bridge of signed differences instead of a funnel, rather than a band wider than the one above it. Offer these edits only when the Stage 2 check showed a plausible candidate; do not invent a target.

Stage 3 — GeoJSON (optional)

Ask with AskUserQuestion: "Bin rows into GeoJSON regions?"

If no, skip to Stage 4 with no geo flags.

If yes:

3a. Check the data can actually be binned

viz smart reaches a region two ways, and only one of them needs coordinates:

• Region key — --locations <col> (with --location-mode) where the column already identifies the region: an ISO-3 country code, a 2-letter US state code, a country name, a GeoJSON feature id, county FIPS/GEOID. No coordinate pair required. This is the path that works against a custom --geojson file, and the feature id must JOIN these values (Stage 3c).

• Place NAME — two different routes, and one precondition that governs both. The precondition: geo.city/town/municipality columns are nominated as region candidates only on a geocode-enabled build (geocodable_name_candidates returns nothing otherwise, and degrades silently). Check qsv --version for geocode before promising either route.

  • Direct, with any --geojson file whose feature ids ARE those names: an ordinary join, no aliases, so Stage 3c's overlap check settles it. examples/viz/nyc_neighborhoods.geojson (keyed by properties.name) is exactly this shape.
  • Alias-based city → county FIPS, --geojson auto/census only — the alias map is synthesized by automatic Census resolution, so a custom file publishes none. viz smart drives this itself: it forward-geocodes implicitly and does not need the --geocode flag, which is why Stage 4 not passing --geocode is no obstacle. (The explicit flag is a viz choropleth concern and is restricted to the iso3/usa-states location modes.)

  So do not reject a name column against a custom file — test the overlap first. Require a region-CODE column only when the names do not join, or when the build has no geocode support.

• Point-in-polygon — --lat/--lon, each row's coordinates tested against the polygons.

Check BOTH before concluding anything, and check the dictionary first — viz smart identifies its region column from the dictionary's concepts, not from header spelling, so $SCHEMA is the authoritative answer and a header regex is only a fallback:

# authoritative: only these geo.* leaves key a polygon. NOT every geo.* concept does -
# geo.latitude/longitude/coordinate_pair/street_address/ip_address/timezone/geonames_id name a
# point or an attribute, and treating them as region keys is the same false positive inverted.
python3 - "$SCHEMA" <<'PROBE'
import json, sys
# mirrors viz.rs REGION_CODE_LEAVES + CITY_NAME_LEAVES
CODE = {"zip_code","zip","postal_code","zcta","census_tract","county","county_fips","state",
        "state_fips","country","country_code","place_fips","fips"}
CITY = {"city","town","municipality"}   # need a geocode-enabled BUILD, not the --geocode flag
props = json.load(open(sys.argv[1])).get("properties", {})
con = {k: str((v.get("x-qsv") or {}).get("concept", "")) for k, v in props.items()}
leaf = lambda c: c.split(".", 1)[1] if c.startswith("geo.") else None
code = [k for k, c in con.items() if leaf(c) in CODE]
city = [k for k, c in con.items() if leaf(c) in CITY]
pair = ([k for k, c in con.items() if leaf(c) == "latitude"],
        [k for k, c in con.items() if leaf(c) == "longitude"])
print("region-code columns :", code or "(none)")
print("city-name columns   :", city or "(none)")
if city:
    print("  ^ nominated as region candidates only on a geocode-enabled BUILD")
    print("    (check `qsv --version` for `geocode`). NOT the --geocode flag,")
    print("    which `viz smart` never needs - see 3a above for the two routes.")
print("lat/lon PAIR        :", "yes" if all(pair) else "no  # a lone lat or lon bins nothing")
PROBE

Do not reach for a region-name regex: it cannot spell every geography (tract, zcta, municipality, town, iso3 all miss a state|county|country pattern), and a false negative here is exactly the mistake this stage used to make. Only when the probe finds neither a region-code column (nor a city/place-name column — which on a geocode-enabled build either joins a custom GeoJSON keyed by those names directly, or resolves to county FIPS on the auto path) nor a complete lat/lon pair does the GeoJSON have no effect — a lone geo.timezone or geo.ip_address is not a region key and does not count — say so then, and offer to proceed without it. A dataset carrying county names (or FIPS codes) and no coordinates at all maps perfectly well, so do not talk the user out of it.

3b. Get the file

Accept a local path, an http(s) URL, or a shortcut name defined in QSV_GEOJSON_SHORTCUTS (a JSON map of name → {path, id}; the shortcut's id supplies --feature-id-key when you don't pass one).

There is also a fourth form, and for US data it is usually the right one: --geojson auto (or census) fetches US county, ZIP Code Tabulation Area, census tract or place boundaries from the Census TIGERweb service, scoped to the states the data names, and sets --feature-id-key to properties.GEOID itself. The user supplies nothing but the CSV — no file to source, and Stage 3c below is unnecessary. Prefer it over hunting for a boundary file. To chart a rate rather than a count, pair it with --denominator census only for county or state maps - Census denominators exist for those two geographies alone, and the fetch hard-errors without a free QSV_CENSUS_API_KEY. For a ZCTA, tract or place layer, get the denominator from the data instead: a measure.population column in the dictionary (describegpt derives x-qsv.denominator from it), or --denominator-key pointing at a boundary property the fetched features carry.

3c. Discover the feature id key — do not guess it

--feature-id-key defaults to id, which is usually wrong. In viz smart's point-in-polygon mode the key labels each binned region, so it must be present on every feature, unique across all of them, and meaningful to a human. Uniqueness alone is not enough: properties.shape_area is perfectly unique and completely useless as a label.

⚠️ That "meaningful to a human" rule is the point-in-polygon rule, where the key LABELS each binned region. On the region-key path it is the wrong test and will cost you the choropleth: there the key must join — its values have to overlap the distinct values of the region column. Pick the key whose values match the CSV (GEOIDs match GEOIDs, names match names), verify the overlap before rendering, and put the human-readable property in --feature-name-key instead, which exists precisely to supply hover labels. Choosing a display name here while the CSV holds GEOIDs resolves nothing and renders no map.

Region-key or name path — ask qsv, do not reimplement the match

qsv viz --check-geojson-key scores every candidate feature-id path against the distinct values of the region column and ranks them by overlap. Use it and take its answer:

qsv viz choropleth "$WORK" --locations "$REGION_COL" --geojson "$GEOJSON" --check-geojson-key

3221 distinct --locations values scored against --geojson 'counties.geojson':
     3221/3221 100.0%  properties.GEOID
        0/3221   0.0%  properties.NAME  (unmatched e.g. 01001, 01003, 01005)

Use: --feature-id-key properties.GEOID

It scores through the same matcher the render path binds with, so a path it reports as a full match will bind at render time — zero-padding (6 vs 06037), ASCII case folding, and the refusal to guess between ambiguous folds (with features CA and ca, the value Ca matches neither) all come out identical by construction. That is why this replaced a hand-written scorer here: a reimplementation that is one tier more generous than viz reports a join you will not get.

It reads the same --geojson sources viz does (local path, http(s) URL, or a QSV_GEOJSON_SHORTCUTS name) and needs no valid --feature-id-key to run — finding one is its job. Candidates include nested paths under properties at ANY depth and top-level foreign members, not just properties.<field>, so a boundary file that keys off either is still scored. Treat a partial match as a warning, not a pass. If nothing matches, the region values and the boundary file disagree, or that GeoJSON cannot key them; say so rather than rendering an empty map.

⚠️ Skip this check entirely when $GEOJSON is an automatic Census spec — auto, census, census:<layer> or either with an @<year> vintage (Stage 3b's fourth form). Every one of them is refused, deliberately. Automatic Census resolution picks the feature-id key itself (properties.GEOID) and prints its own region coverage, so there is nothing to discover; and a place-NAME column binds through an alias map the check does not model, which would score every candidate at 0% on a setup that renders correctly. Go straight to Stage 4 and read the coverage line viz reports.

Point-in-polygon path — rank by uniqueness and readability

There is no join to test on this path (rows are binned by geometry), so the question is which property makes a good label. That is a judgement about the boundary file alone, which the script below answers.

It mirrors exactly one rule from viz: build_pip_features skips features without Polygon/MultiPolygon geometry, so the script ranks over those features only. Rank over every raw feature instead and a skipped point that duplicates an id makes that id look non-unique, which reports a file as unkeyable on a key that would have labelled every binned region. That one geometry check is the only parity this script needs — the match tiers it used to reimplement now live behind --check-geojson-key above.

It accepts the same file source forms --geojson does — a local path, an http(s) URL, or a QSV_GEOJSON_SHORTCUTS name. If you only handle local paths here, a URL or shortcut fails at discovery even though viz would have accepted it.

python3 - "$GEOJSON" <<'PY'
import json, sys, re, os, collections, urllib.request

def load_geojson(src):
    """Local path, http(s) URL, or a QSV_GEOJSON_SHORTCUTS name.

    Mirror viz's resolution order (src/cmd/viz.rs resolve_and_validate_geojson): an
    http(s) URL or an EXISTING local file is a direct source; only a value that is
    neither is looked up as a shortcut NAME. This keeps a local file whose name
    collides with a shortcut loading as the file (as viz does), and it never lets
    a malformed QSV_GEOJSON_SHORTCUTS break a direct file/URL input.
    """
    hint = None
    is_url = src.startswith(("http://", "https://"))
    if not is_url and not os.path.isfile(src):
        raw = os.environ.get("QSV_GEOJSON_SHORTCUTS")
        if not raw:
            sys.exit(f"--geojson '{src}' is not an existing file or http(s) URL, "
                     "and QSV_GEOJSON_SHORTCUTS is not set")
        shortcuts = json.loads(raw)           # invalid JSON surfaces as an error
        if src not in shortcuts:
            sys.exit(f"unknown --geojson shortcut '{src}'; "
                     f"defined: {', '.join(sorted(shortcuts)) or '(none)'}")
        entry = shortcuts[src]
        hint = entry.get("id")                # shortcut may carry its own id key
        src = entry["path"]
        is_url = src.startswith(("http://", "https://"))
    if is_url:
        with urllib.request.urlopen(src, timeout=30) as r:
            return json.loads(r.read().decode("utf-8")), src, hint
    with open(src) as fh:
        return json.load(fh), src, hint

g, resolved, hint = load_geojson(sys.argv[1])
feats = g.get("features", [])
if not feats:
    sys.exit("no features")
# Rank over the features viz will actually BIN. build_pip_features skips anything without
# Polygon/MultiPolygon geometry, so ranking over every raw feature reports a key as non-unique
# whenever a skipped point duplicates it - and then declares the file unusable on a key that
# would have labelled every binned region perfectly.
feats = [f for f in feats
         if (f.get("geometry") or {}).get("type") in ("Polygon", "MultiPolygon")]
if not feats:
    sys.exit("no Polygon/MultiPolygon features - viz cannot bin rows into this file")
print(f"source: {resolved}")
if hint:
    print(f"shortcut supplies --feature-id-key {hint} (override below if you prefer)")

# Geometry-derived / bookkeeping fields: unique, but meaningless as a region label.
NOISE = re.compile(r"shape|area|leng|length|perim|acres|sqmi|aland|awater|"
                   r"intptlat|intptlon|^lat|^lon|_x$|_y$|"
                   r"date|time|edited|created|updated|version", re.I)

def floatish(v):
    return isinstance(v, float) or (isinstance(v, str) and re.fullmatch(r"[+-]?\d+\.\d+", v.strip()))

cands = collections.defaultdict(list)
for f in feats:
    if f.get("id") is not None:
        cands["id"].append(f["id"])
    for k, v in (f.get("properties") or {}).items():
        if isinstance(v, (str, int, float)):
            cands[f"properties.{k}"].append(v)

good, other = [], []
for key, vals in cands.items():
    if len(vals) != len(feats):                    # missing on some feature
        continue
    if len(set(map(str, vals))) != len(feats):     # not unique
        continue
    demote = bool(NOISE.search(key)) or all(floatish(v) for v in vals)
    (other if demote else good).append((key, vals[:3]))

def show(title, rows):
    print(f"\n{title}")
    if not rows:
        print("  (none)")
    for key, sample in rows:
        print(f"  {key:<32} e.g. {sample}")

print(f"{len(feats)} usable (polygon) features")

show("RECOMMENDED feature-id-key (unique, meaningful):", good)
show("Unique but geometry/bookkeeping - avoid:", other)
if not good and not other:
    print("\nNo property is unique across all features. This GeoJSON cannot key regions as-is.")
PY

What you offer via AskUserQuestion depends on the path Stage 3a identified:

• Region key or name path — do not offer this ranking at all. Take --check-geojson-key's highest scorer, ideally a full match. Readability is irrelevant here and actively misleading: a numeric properties.GEOID/OBJECTID that joins is correct, while a pretty properties.hood that joins nothing renders an empty choropleth.
• Point-in-polygon path — offer the RECOMMENDED keys and favour a short region code or name (properties.nta2020, properties.hood) over a surrogate key (properties.OBJECTID, a GUID): here the value really does label each binned region.

If nothing is unique, say so plainly: the GeoJSON cannot key regions as-is.

Then pick --feature-name-key (e.g. properties.name) for human-readable hover labels — that is where a readable property belongs. Stage 4 passes it only when you set FEATURE_NAME_KEY.

Stage 4 — Render

Ask for --dataset-pid with AskUserQuestion (a persistent identifier — a DOI, ARK, Handle, or a URL). It is optional; allow the user to skip it.

qsv viz smart "$WORK" \
  --smarter --bivariate \
  --dictionary "$SCHEMA" --dict-info \
  ${GEOJSON:+--geojson "$GEOJSON"} \
  ${FEATURE_ID_KEY:+--feature-id-key "$FEATURE_ID_KEY"} \
  ${FEATURE_NAME_KEY:+--feature-name-key "$FEATURE_NAME_KEY"} \
  ${DATASET_PID:+--dataset-pid "$DATASET_PID"} \
  -o "$OUT"

• --smarter runs qsv moarstats --advanced first, enriching the stats cache with distribution shape (bimodality, entropy, skewness, outlier share, Gini — the last unlocks Lorenz curves for the most unequal additive measures). Costs one extra pass and writes <stem>.stats.csv + sidecars + .idx. It applies only under default parsing: --no-headers or a custom --delimiter silently falls back to the standard Data Schematic.
• --bivariate adds a normalized-mutual-information heatmap plus — only when there are more than 8 chartable columns — a ranked "top relationships" bar. It implicitly turns on --dictionary infer when --dictionary is not set — so passing $SCHEMA explicitly is what stops viz from calling the LLM a second time. Never pass --bivariate without a dictionary in this workflow. Capped at 50 columns; wider datasets skip both panels with a warning.
• --dict-info embeds the dictionary in a side drawer next to the plots, adds an info icon per panel, and a "Data Dictionary" link under the title. The drawer also carries download buttons for the sidecars this run actually read — the schema, the charted frequency counts, the stats cache + metadata, and the bivariate CSV — all bundled into the HTML, so a recipient needs no access to your machine. Absolute local paths are stripped from the embedded metadata (sharing a Data Schematic does not disclose your directory layout); sidecars over 4 MB are skipped with a note. HTML only — ignored with a note when exporting an image.
• -o must end in .html. An image extension (.png, .svg, …) silently switches viz to the static-export path, which needs a browser/webdriver and drops --dict-info.

The data viewer drawer (--preview-threshold, default 50000)

Independent of --dictionary/--dict-info: an (Explore) link beside the row count in the metadata table opens the underlying rows in a searchable bottom drawer. Every row is embedded while the dataset has at most <n> rows; above that only the first <n> are, and the link reads (Preview).

This is the one flag here with a real cost: embedded rows grow the HTML — and the reader's browser memory — in proportion to rows × columns. Tell the user the size (Stage 5 prints it) rather than letting them discover it. Lower the threshold, or pass --preview-threshold 0 to drop the viewer entirely, when the Data Schematic is meant to be emailed around.

--photos — ask first, never enable silently

If a column holds image URLs, --photos makes dwelling on a map point reveal that row's photo. It is off by default and deliberately so: images load from whatever third-party host the data names, so every person who opens the Data Schematic requests those URLs directly and reveals their IP to that host. Only pass it if the user asks for it after being told that. HTML tile-map panel only.

Stage 5 — Verify, then report

Never claim success without checking. viz smart prints what it skipped to stderr — surface that to the user verbatim; it is the most useful line it emits.

test -s "$OUT" || { echo "no Data Schematic written"; exit 1; }
python3 - "$OUT" <<'PY'
import re, sys
h = open(sys.argv[1], encoding="utf-8", errors="replace").read()
print(f"{len(h)/1e6:.1f} MB")
print("dictionary drawer embedded:", "qsv-dict-drawer" in h)
print("dictionary back-links:", h.count("View chart"))
m = re.search(r"Data — [^\"<]{0,60}", h)
print("data viewer:", m.group(0) if m else "disabled / not embedded")
print("guided tour config:", "qsv-tour-config" in h)
print("tour narration in dict drawer:", 'class="qsv-dict-tour"' in h)
PY

View chart counts the dictionary's back-links to panels, not the panels themselves — viz emits one only where a matching panel element exists. Use the stderr note for what was drawn and skipped; that is authoritative.

Then tell the user:

• which columns denull cleaned, and how many cells were blanked
• how many columns got a role/concept from the dictionary
• which columns viz smart skipped, and why (its stderr note names them)
• whether the KPI row, any gauge tile, and the pipeline panel rendered — and if a hint from Stage 2 was dropped, viz says why on stderr (a gauge_range whose range excludes the data, a pipeline naming a missing column)
• if a region-level column (a population/area, or anything a region names as its x-qsv.denominator) has no KPI tile, that is by design, not a regression: its value repeats per region, so a dataset-wide total would be a multiple of the real one and viz omits the tile rather than print it (issue #4528). Any grouped bar SELECTED for that column is collapsed per region — but do not promise one: viz smart draws at most a single measure-by-dimension panel (the strongest measure/dimension pair, and only above an eta-squared threshold), so a region-level column frequently has no bar of its own. Report what actually rendered. Say so either way — the missing tile is otherwise read as a bug
• the data viewer's state: all rows (Explore) or a truncated preview, and what it costs in file size
• the GeoJSON coverage note, if any (points that fell outside every region)
• the path to $OUT

If the user can open a browser, offer to render it. Do not assert the Data Schematic "looks right" — you cannot see it. Unless you have a browser-automation MCP — then Stage 6 lets you.

Stage 6 — Tour refinement (optional, browser)

The x-qsv.tour narration from Stage 2 was written blind: the LLM never saw which panels viz smart actually drew. With a browser you can close that loop — step through the Tour, judge each narration against what is really on screen, and refine the schema. Skip this stage (and say so) when no browser-automation MCP is available.

Tool-agnostic: use whatever browser-automation MCP is available — Playwright MCP, claude-in-chrome, or any other. Every check below is specified by selector/anchor, not by tool.

1. Open file://$OUT (or serve it locally if the tool requires http).
2. Read the resolved tour from the element #qsv-tour-config — its JSON payload lists the steps this page actually built: which step ids exist, the prose each carries, and for each panel step its key — the stable raw-field-name or @kind token that x-qsv.tour.panels/panel_order address. Use key, never the display title (titles are decorated: "activated (right-skewed)"). This is ground truth; the schema's overrides/panels only applied where a matching step/panel exists.
3. Replay the tour. It auto-runs only on first visit — click the Tour pill in the header, or remove every localStorage key starting with qsv-viz-tour-seen- (the key is suffixed with a page hash and the pathname, so clear by prefix) and reload.
4. Step through and judge each popover against three things: (a) the audience's register, (b) what is actually visible on that panel — narration must never reference a chart that wasn't drawn or numbers that aren't shown, and (c) the collapsed <details class="qsv-dict-tour"> section on the dictionary page (open the drawers with the page's qsvOpenDict / qsvOpenData links as the tour does).
5. Refine $SCHEMA with an inline python3 JSON merge — load, mutate ONLY s["x-qsv"]["tour"], dump. Never sed/regex the file, never touch other keys. Unlike the Stage 2 LLM pass, you can now see which panels rendered, so you MAY also set @kind-token panels entries (@kpi, @correlation, @timeseries, @map, @choropleth, @scatter, …) and a panel_order array to pick and order the spotlights (viz caps them at the --tour-steps budget, default 8). Keep version the integer 1; keep language BCP-47 matching the page locale.
6. Re-render and re-verify. Stage 4 passes --dictionary "$SCHEMA" by path, so re-running it is cheap (no LLM call, no sidecar-reuse trap). Re-open the page and spot-check the changed steps. At most two refinement loops — then report what changed and stop.

Guardrails

• Never run denull --apply with -o pointing at the input, and never with - (stdin). It refuses both, but don't rely on that.
• If denull confirms nothing, do not create a .denulled.csv. An empty transform step is noise.
• Never hand-write the JSON Schema. It comes from describegpt. role, concept, title, description and x-qsv.aggregation are adjusted only through the Stage 2.5 edit_dictionary.py TUI — never by editing the JSON by hand (an off-vocab role/concept typed into the raw file silently routes a column to the wrong panel; the TUI validates against the vocab and flags drift). The exceptions are the six keys the TUI does not own — x-qsv.gauge_range, x-qsv.target, x-qsv.currency, x-qsv.denominator and the dataset-level x-qsv.relationships and x-qsv.tour — which qsv documents as hand-edited. x-qsv.tour is freeform prose (nothing for a validator to check), but three of its fields are load-bearing: version must stay the integer 1, language must stay BCP-47 matching the page locale, and prose is plain text only — never HTML or Markdown. When editing it, never touch any other key in the schema.
• The Stage 2.5 TUI is out-of-band: it needs the user's real terminal. Never try to launch it through your Bash tool and "drive" it — that shell is not a TTY, and the script will refuse. Print the command, wait, then re-read.
• --dictionary infer runs describegpt without --infer-null-values. If you want null sentinels in the dictionary, you must generate it yourself and pass the path.
• Statistics over cleaned columns are complete-case: they describe the rows that have a value. denull makes the missingness visible; it does not make it ignorable. Do not reach for qsv stats --nulls to "restore" the blanks — that imputes zero for the mean-family statistics while the median and quartiles keep ignoring them, so the summary stops agreeing with itself.

Example

/visual-data-dictionary NMBGMRManualWaterLevels.csv

1. denull confirms 6 columns (HoleDepth, WellDepth, CasingDiameter, CasingDepth, DepthToWaterBGS, DataQuality), blanks 8,278 cells; all 6 promote from String to Integer/Float.
2. describegpt writes NMBGMRManualWaterLevels.denulled.schema.json with role/concept on 25/25 columns.
3. User declines GeoJSON (the file has UTM Easting/Northing, not lat/lon).
4. viz smart --smarter --bivariate --dict-info writes NMBGMRManualWaterLevels.html, charting the numeric columns and skipping _id / PointID (identifiers) and the date columns (which feed the time-series panel instead). Report the counts viz actually prints on stderr — panel selection moves with each release, so never quote a remembered number.

Before cleaning, viz smart skipped 11 columns and warned that 5 of them looked like numeric data held back by a literal NULL. That warning is the reason Stage 1 exists.

A GeoJSON run reports its binning coverage on stderr — pass it on verbatim:

viz smart: 54 of 409 points were snapped to the nearest region
           (cap 0.24 km, auto-derived from region size and coordinate precision)

denull finding nothing is a normal outcome, not a failure. Say so and move on.