swe-bench
GitHub在官方 Docker 镜像中运行 SWE-bench 实例，自动拉取镜像、启动容器并执行 OpenSquilla Agent 生成补丁。支持验证和多语言数据集评估，提供详细的日志、成本及结果解析，适用于基准测试或代码修复任务。
Trigger Scenarios
Install
npx skills add opensquilla/opensquilla --skill swe-bench -g -y
SKILL.md
Frontmatter
{
"name": "swe-bench",
"metadata": {
"platform": {
"emoji": "🧪",
"install": [
{
"id": "swebench-extra",
"kind": "uv",
"label": "Install SWE-bench extras (uv pip)",
"package": "opensquilla[swebench]"
}
],
"requires": {
"env": [
"OPENROUTER_API_KEY"
]
}
}
},
"triggers": [
"swe-bench",
"swebench",
"SWE-bench",
"跑一道题",
"解一道 SWE",
"benchmark instance"
],
"provenance": {
"origin": "opensquilla-original",
"license": "Apache-2.0",
"maintained_by": "OpenSquilla"
},
"description": "Run SWE-bench instances with an OpenSquilla agent inside the official Docker images. Trigger when the user wants to run\/solve\/evaluate a SWE-bench instance (e.g. 'run django__django-16429', 'test OpenSquilla on SWE-bench', '跑一道 SWE-bench 题'), benchmark the agent on SWE-bench_Verified or SWE-bench_Multilingual, or check whether a generated patch resolves an instance. Optional dependency — install via `pip install opensquilla[swebench]`; also needs the docker CLI and an OPENROUTER_API_KEY."
}
swe-bench
Run a SWE-bench instance end-to-end: ensure the instance's Docker image is available (pull from Docker Hub if missing), start a container, run an OpenSquilla agent against the issue, collect the patch, and optionally run the official evaluation.
Prerequisites — guide the user, don't dead-end
SWE-bench mode runs the official evaluation images, so it needs the Docker
CLI. Docker is NOT a hard gate on this skill: if it is missing, the
opensquilla swebench command prints the exact install command for the
user's OS and exits. When that happens, relay the install guidance to the
user ("SWE-bench needs Docker — install it with ..., then I can run this")
instead of saying the task is impossible. Also mention that solving a
real-repository coding task (not a benchmark instance) does NOT need
Docker — that is what the code-task skill is for.
Commands
Solve one instance (auto-pulls the image when missing):
opensquilla swebench solve <instance_id> --dataset verified --json
--datasetacceptsverified,multilingual, or a full HuggingFace dataset name.- Add
--evaluateto run the official harness afterwards and report whether the patch actually resolves the instance (resolvedin the JSON output). - Add
--model <model>/--thinking <level>to pin a model; leave them off to let squilla_router decide. --timeout <seconds>defaults to 1200.
Pre-fetch an image only:
opensquilla swebench pull <instance_id>
Evaluate an existing predictions file:
opensquilla swebench eval <predictions.jsonl> --dataset verified
Reading the result
solve --json prints one JSON object: state (patch_collected is
success), patch_path, artifact_dir, resolved (true/false, or null
when --evaluate was not used), duration_seconds, usage
(cost/tokens), error.
The full artifact trail (prompt, agent log, transcript, usage, patch)
lives under artifact_dir.
What to tell the user
- Before starting: if the image is not local yet, warn that the first run pulls 1-3 GB and can take a few minutes; a solve typically takes 10-30 minutes depending on the instance and timeout.
- After finishing: report
state, whether the patch is non-empty,resolvedwhen evaluation ran, the cost fromusage, and thepatch_pathso the user can inspect the diff. - On
failed/timeout: quote theerrorfield and point at<artifact_dir>/agent_stdout.logfor the full trace.
Constraints
- Images are x86_64; on ARM hosts only locally pre-built images work.
- The run happens on this machine (the gateway host) — docker must be reachable from here, and disk usage grows with each pulled image.
- Long runs block the command; for batches beyond a handful of instances, suggest the user run the CLI directly in tmux instead of going through chat.
Version History
- 7f72a32 Current 2026-07-05 18:40
