Agent Skills
› xberg-io/xberg
› benchmark-workflow
benchmark-workflow
GitHub管理基准测试工作流，确保Ground-truth完整性，诊断运行失败原因，规范评估指标与缓存处理。
Trigger Scenarios
执行提取基准测试
评估质量分数
管理基准测试夹具或真相数据
分析基准测试结果
Install
npx skills add xberg-io/xberg --skill benchmark-workflow -g -y
SKILL.md
Frontmatter
{
"name": "benchmark-workflow",
"type": "Playbook",
"paths": [
"tools\/benchmark-harness\/**"
],
"title": "Benchmark Workflow",
"x-ai-rulez": {
"id": "benchmark-workflow",
"kind": "skill",
"metadata": {
"name": "benchmark-workflow"
}
},
"description": "Run extraction benchmarks; score quality and manage fixtures or ground truth."
}
Benchmark workflow
The benchmark system lives in tools/benchmark-harness/; the GitHub workflow is
.github/workflows/benchmarks.yaml. The workflow is dispatch-only, so it does not run on push or gate merges. Treat
a result as evidence for its exact commit SHA and inputs, not for newer local work.
Ground-truth integrity
- Never use Xberg's own extractor output as benchmark ground truth. Use an independent source and record it in the
fixture's
ground_truth.sourcefield (manual,vision,pdf_text_layer,pandoc,python-docx, and similar). - Before blaming ground truth for a score, render or otherwise inspect the source document. If the derived
.mdor.txtdisagrees with the source, fix the ground truth; if it agrees, investigate the extractor or metric. - Use the fixture schema in
tools/benchmark-harness/README.md, the generator attools/benchmark-harness/scripts/generate_markdown_gt.py, and the harnessvalidate-gtcommand implemented intools/benchmark-harness/src/validate_gt.rs. Do not replace these with an ad-hoc conversion pipeline. - A quality claim requires the same corpus, config, renderer, cache state, and metric on control and experiment. Disable or invalidate extraction and OCR caches before A/B runs whose output behavior changed.
Diagnosing runs
- Separate infrastructure failures from extraction or quality failures. A missing backend library, absent fixture, malformed artifact, or runner setup error does not describe extractor quality.
- Inspect the per-adapter artifacts before the aggregate job. Aggregate contract failures may be caused by missing or unexpectedly named artifacts even when individual adapters ran.
- Compare accepted OCR pages before raw word counts. Rejected OCR pages contribute neither text nor structured paragraphs.
- Measure headings and lists using Markdown output. Plain output normalizes away list markers and cannot distinguish detection from rendering.
- Do not quote a coverage, latency, or quality threshold unless the workflow or harness currently enforces it.
When changing harness behavior, add focused tests for the report or artifact contract and run the task that exercises the affected adapter before dispatching the remote workflow.
Version History
-
844bb91
Current 2026-11-03 09:36
重构技能路由机制，按路径范围限制技能可见性；缩短技能描述以节省上下文预算。
- 3be78fd 2026-10-19 16:30
- d8e4815 2026-08-28 18:30
