Agent Skills › ory/lumen › add-benchmark

add-benchmark

GitHub

从GitHub Issue或PR提取真实Bug修复，构建SWE基准测试任务。处理URL验证、仓库筛选、代码克隆、补丁生成及自动化校验，最终输出结构化任务数据供基准流水线使用。

.claude/skills/add-benchmark/SKILL.md ory/lumen

触发场景

用户提供GitHub Issue或PR链接并请求添加到基准测试需要为特定语言创建新的SWE基准测试用例

安装

npx skills add ory/lumen --skill add-benchmark -g -y

更多选项

非标准路径

npx skills add https://github.com/ory/lumen/tree/main/.claude/skills/add-benchmark -g -y

不安装直接使用

npx skills use ory/lumen@add-benchmark

指定 Agent (Claude Code)

npx skills add ory/lumen --skill add-benchmark -a claude-code -g -y

安装 repo 全部 skill

npx skills add ory/lumen --all -g -y

预览 repo 内 skill

npx skills add ory/lumen --list

SKILL.md

Frontmatter

{
    "name": "add-benchmark",
    "description": "Add a new SWE benchmark task from a real GitHub bug-fix. Use when the user provides a GitHub issue or PR URL and wants to add it to the bench-swe pipeline.",
    "argument-hint": "<github-issue-or-pr-url> <language>",
    "disable-model-invocation": true
}

Add SWE Benchmark

Add a new benchmark task to the bench-swe pipeline from a real GitHub bug-fix. The human provides the GitHub issue or PR URL; the agent handles extraction, validation, and file creation.

Arguments

url (required): GitHub issue or PR URL (e.g. https://github.com/gorilla/mux/issues/534 or https://github.com/gorilla/mux/pull/585)
language (required): One of: go, python, typescript, javascript, rust, ruby, java, c, cpp, php, csharp

Repository selection criteria

Good benchmark repos are focused libraries with a clear bug — not large applications. Before submitting a URL, prefer repos that are:

Size: < 50 MB and < 800 source files (excludes vendor/node_modules)
Dependencies: < 50 direct dependencies (go.mod, package.json, etc.)
Scope: a library or small service, not a monorepo or full application

The agent will reject repos that exceed these limits.

Steps

Dispatch the task-curator agent with the provided arguments. The agent will:
- Validate inputs (URL, language)
- Check repository size and dependency count (rejects oversized repos)
- Resolve the fix PR (from issue or directly)
- Clone the repo, extract base/fix commits, and generate the gold patch
- Determine the test command from repo conventions
- Write task JSON to bench-swe/tasks/{language}/ and patch to bench-swe/patches/
- Run 5 inline verification checks (patch applies, files match, no leaks, schema completeness, no test files in patch)
- Fix any issues found during verification
Report the result including:
- Task ID, repo, issue URL
- Files and lines changed
- Verification table