Agent Skills › ory/lumen › add-benchmark

add-benchmark

GitHub

用于将真实GitHub缺陷转换为SWE基准测试任务。接收URL和语言参数，自动验证仓库规模，调用子代理提取代码差异、生成补丁并执行多项验证检查，最终输出标准化的基准数据。

.claude/skills/add-benchmark/SKILL.md ory/lumen

Trigger Scenarios

用户提供GitHub Issue或PR链接并要求添加到基准测试需要将软件缺陷修复案例转化为自动化测试用例

Install

npx skills add ory/lumen --skill add-benchmark -g -y

More Options

Non-standard path

npx skills add https://github.com/ory/lumen/tree/main/.claude/skills/add-benchmark -g -y

Use without installing

npx skills use ory/lumen@add-benchmark

指定 Agent (Claude Code)

npx skills add ory/lumen --skill add-benchmark -a claude-code -g -y

安装 repo 全部 skill

npx skills add ory/lumen --all -g -y

预览 repo 内 skill

npx skills add ory/lumen --list

SKILL.md

Frontmatter

{
    "name": "add-benchmark",
    "description": "Add a new SWE benchmark task from a real GitHub bug-fix. Use when the user provides a GitHub issue or PR URL and wants to add it to the bench-swe pipeline.",
    "argument-hint": "<github-issue-or-pr-url> <language>",
    "disable-model-invocation": true
}

Add SWE Benchmark

Add a new benchmark task to the bench-swe pipeline from a real GitHub bug-fix. The human provides the GitHub issue or PR URL; the agent handles extraction, validation, and file creation.

Arguments

url (required): GitHub issue or PR URL (e.g. https://github.com/gorilla/mux/issues/534 or https://github.com/gorilla/mux/pull/585)
language (required): One of: go, python, typescript, javascript, rust, ruby, java, c, cpp, php, csharp

Repository selection criteria

Good benchmark repos are focused libraries with a clear bug — not large applications. Before submitting a URL, prefer repos that are:

Size: < 50 MB and < 800 source files (excludes vendor/node_modules)
Dependencies: < 50 direct dependencies (go.mod, package.json, etc.)
Scope: a library or small service, not a monorepo or full application

The agent will reject repos that exceed these limits.

Steps

Dispatch the task-curator agent with the provided arguments. The agent will:
- Validate inputs (URL, language)
- Check repository size and dependency count (rejects oversized repos)
- Resolve the fix PR (from issue or directly)
- Clone the repo, extract base/fix commits, and generate the gold patch
- Determine the test command from repo conventions
- Write task JSON to bench-swe/tasks/{language}/ and patch to bench-swe/patches/
- Run 5 inline verification checks (patch applies, files match, no leaks, schema completeness, no test files in patch)
- Fix any issues found during verification
Report the result including:
- Task ID, repo, issue URL
- Files and lines changed
- Verification table