Leviathan: search large datasets without reading them

The leviathan CLI answers questions from an indexed copy of a dataset. One call returns a handful of ranked records, usually under 1,000 tokens, however large the data is. Prefer it to grepping, cat, or paging through exports: those cost tokens in proportion to the data.

First: learn the dataset

leviathan describe

This shows what one record and one group are called (ticket/customer, event/host, ...), the fields that are searched, the filter fields with their common values, the date range, and example calls. Run it once per session before searching, and use its exact field names in --where.

An index may hold several kinds of record (tickets, customers, orders). describe lists them; ids read <kind>:<id>; --where _kind=<kind> keeps one kind and aggregate --by _kind counts each. A customer's group can hold their tickets and orders and the customer record itself.

Search

leviathan search "<words as the user said them>"                      # ranked, everything
leviathan search -g "<group as the user said it>" "<words>"           # within one customer/host/machine
leviathan search "<words>" --where status=open --where priority=high  # exact filters (same field twice = either)
leviathan search "<words>" --since 2024-01 --until 2024-06            # date range, inclusive
leviathan search '"exact phrase" -excluded'                           # quote the whole query for -exclusions
leviathan search -g acme "<words>" --scope others                     # same problem, every other group
leviathan recent -g "<group>" -n 10                                   # newest first, no words needed
leviathan resolve "<group name>"                                      # which groups match a name
leviathan get <id> [<id> ...]                                         # complete records

Add --json for structured output, -n for more results (max 50), and --offset to page. The index comes from --index or LEVIATHAN_INDEX (default ./leviathan.db).

Count and total

For "how many", "how much" and "which ... most", aggregate instead of reading or scanning the data. It is exact over every matching record and takes the same group, filters, words and dates as search:

leviathan aggregate -g "<group>" --where status=open --since 2024           # how many
leviathan aggregate --by group --where kind=breakdown --since 2023 --until 2023  # which group has the most
leviathan aggregate --by month -g "<group>"                                 # per month (or year, day)
leviathan aggregate --by priority --group-contains "Hall G"                 # per filter value, groups by name words
leviathan aggregate --metric sum --field hours -g "<group>" --since 2025    # sum, avg, min or max of a field
leviathan aggregate --metric distinct --field "labor[].tech"                # how many different values
leviathan aggregate --by "parts[].name"                                     # per value of any field, not just filters
leviathan aggregate --by group,month --since 2025                           # two at once
leviathan aggregate --by crew --numeric "hours>4"                           # only records meeting number conditions

Reading the output

• The header states the scope and shown N of M. Zero means none found, not none exist: retry with fewer or different words, or drop a filter.
• Each result has an id, its group, a date, the configured display fields, and a match: line from the text that matched.
• Results marked OTHER <GROUP> come from a different customer, host or machine than the one asked about. Say so, and keep each result's group visible.

Exit status

0 ok (including zero hits) · 1 error · 2 bad request (unknown filter field, bad date) · 3 group unknown or ambiguous. On 3, show the listed candidates and ask which one the user means. Never guess the group.

No index yet?

leviathan index <dir>                                    # everything in it, a kind per dataset, mappings inferred
leviathan init <files|dir|db.sqlite> -o leviathan.toml   # or: propose a field mapping, with reasons
# review and edit leviathan.toml (id, title, text, group, date, filters)
leviathan index                                          # reads source.paths from leviathan.toml
leviathan describe

If you already know the schema, write leviathan.toml directly; see docs/CONFIG.md in the Leviathan repository. Databases are indexed from their CLI exports through stdin, for example psql -At -c "SELECT row_to_json(t) FROM t" | leviathan index - -c leviathan.toml.