Agent Skills › dathere/qsv › data-quality

data-quality

GitHub

提供表格数据质量评估的快速参考维度与修复决策树，涵盖完整性、一致性等指标及对应命令，指导用户进行数据清洗和异常处理。

.claude/skills/skills/data-quality/SKILL.md dathere/qsv

Trigger Scenarios

需要快速检查表格数据质量问题发现数据异常需选择修复命令了解数据质量评估维度

Install

npx skills add dathere/qsv --skill data-quality -g -y

More Options

Non-standard path

npx skills add https://github.com/dathere/qsv/tree/master/.claude/skills/skills/data-quality -g -y

Use without installing

npx skills use dathere/qsv@data-quality

指定 Agent (Claude Code)

npx skills add dathere/qsv --skill data-quality -a claude-code -g -y

安装 repo 全部 skill

npx skills add dathere/qsv --all -g -y

预览 repo 内 skill

npx skills add dathere/qsv --list

SKILL.md

Frontmatter

{
    "name": "data-quality",
    "description": "Quality dimensions quick reference and remediation decision tree for tabular data assessment"
}

Data Quality Assessment with qsv

For the full step-by-step profiling workflow, use the /data-profile command. This skill provides quick-reference guidance for quality assessment and remediation decisions.

Quality Dimensions (Quick Reference)

Dimension	Key Question	Primary Check	Red Flag
Completeness	Missing values?	`stats` — `nullcount`, `sparsity`	Sparsity > 0.5
Uniqueness	Unwanted duplicates?	`stats --cardinality` vs row count	Key column cardinality < row count
Validity	Correct formats/types?	`stats` — `type`; `validate schema.json`	String type on numeric column
Consistency	Uniform formats?	`frequency` — case variants; `sniff` — encoding	Same value in different cases
Accuracy	Plausible values?	`stats` — min/max/stddev	Values > 3 stddev from mean
Column Name Quality	Headers safe & descriptive?	`safenames --verify`	Spaces, special chars, or duplicates in headers
Conformity	Values follow standards?	`searchset` with domain regex	Non-standard codes (country, state, zip, phone)
Referential Integrity	Foreign keys valid?	`joinp --left-anti`	Orphaned references across related files
Injection Safety	Malicious payloads?	`searchset` with injection regex	Formula/SQL injection patterns in cells
Documentation	Dataset described?	`describegpt --all`	No Data Dictionary or Description

Remediation Decision Tree

When a quality issue is found, choose the right fix:

Problem	Severity	Fix Command	When to Skip
Ragged rows	High	`fixlengths`	Never — breaks downstream tools
Wrong encoding	High	`input`	File is already UTF-8 (check with `sniff`)
Unsafe column names	Medium	`safenames`	Headers already safe (no spaces/special chars)
Leading/trailing whitespace	Medium	`sqlp` with `TRIM(col)`	Stats show no difference between `min`/`max` lengths and trimmed values
Duplicate rows	Medium	`dedup` (or `extdedup` for >1GB)	`stats --cardinality` on key columns shows all unique
Inconsistent case	Low	`sqlp` with `UPPER(col)` or `LOWER(col)`	`frequency` shows no case variants
Empty values	Low	`sqlp` with `COALESCE(NULLIF(col, ''), 'N/A')`	Nulls are semantically meaningful
Non-conforming values	Medium	`searchset` + `search --flag`	No domain standard applies
Orphaned foreign keys	Medium	`joinp --left-anti`	Single-file dataset with no references
Injection payloads	High	`searchset` with injection regex + sanitize	Data is internal-only and never opened in spreadsheets or loaded into databases
Invalid rows	Low	`validate schema.json` + filter	No schema available

Fix Ordering

Always apply fixes in this order to avoid cascading issues:

1. input          (encoding — must be UTF-8 before anything else)
2. safenames      (headers — fixes names before column references)
3. fixlengths     (structure — ensures consistent field counts)
4. sqlp with TRIM()    (whitespace — clean values before dedup)
5. dedup          (duplicates — remove after trimming so "foo " and "foo" match)
6. validate       (validation — check against schema last)

Stats Cache as Quality Data Schematic

After running stats --cardinality --stats-jsonl (basic moarstats auto-runs), read the .stats.csv cache to assess quality in one pass:

Cache Column	Quality Signal
`nullcount`	Completeness — 0 is ideal
`sparsity`	Completeness — ratio of nulls (0.0–1.0)
`cardinality`	Uniqueness — compare to row count
`type`	Validity — check expected types
`min` / `max`	Accuracy — plausible range?
`mean` / `stddev`	Accuracy — outlier detection (>3σ)
`outliers_total_cnt`	Accuracy — from moarstats; outlier count per column
`mode`	Consistency — dominant value expected?

Advanced Stats (via `moarstats --advanced`)

Run moarstats --advanced to enrich the cache with distribution shape metrics:

Cache Column	Quality Signal
`kurtosis`	>3 heavy tails (outlier-prone), <3 light tails; >10 = extreme outliers
`bimodality_coefficient`	>=0.555 suggests bimodal distribution (possible mixed populations)
`jarque_bera_pvalue`	<0.05 = NOT normally distributed; flag analyses assuming normality
`gini_coefficient`	Near 1 = extreme concentration; near 0 = uniform
`shannon_entropy`	Low = concentrated values; high = diverse
`winsorized_mean`	Compare to `mean` — large difference signals outlier influence
`median_mean_ratio`	<0.8 or >1.2 = significantly skewed; mean may be misleading
`range_stddev_ratio`	Very high = extreme outliers relative to variability
`cv`	>100% = high relative variability; data is highly spread relative to mean
`mad_stddev_ratio`	>0.8 = stddev is reliable; <<0.8 = outliers inflating stddev
`mode_zscore`	Far from 0 = mode is atypical; possible mixed populations
`trimean`	Robust central tendency: (Q1 + 2*median + Q3)/4; compare to mean for skew detection
`midhinge`	Midpoint of middle 50%: (Q1+Q3)/2; robust center measure
`robust_cv`	MAD/abs(median); outlier-resistant coefficient of variation
`theil_index`	Inequality measure (0=equal); decomposable into within/between group; only for positive values
`mean_ad`	Average absolute distance from mean; less sensitive to outliers than stddev
`simpsons_diversity_index`	Probability two random values differ (0-1); more intuitive than entropy