Agent Skills
› NeverSight/learn-skills.dev
› run-status-monitor
run-status-monitor
GitHub用于轻量级查询实验运行状态（如排队、卡顿、进度及预计结束时间）。支持本地、SLURM、RunAI或SSH环境，通过探针压缩日志并生成摘要报告。禁止用于启动新任务、调试错误或分析意外结果，需遵循资源占用监控与环境开销评估规范。
Trigger Scenarios
查询实验当前进度
判断Job是否卡住
获取预计剩余时间
查看中间结果摘要
Install
npx skills add NeverSight/learn-skills.dev --skill run-status-monitor -g -y
SKILL.md
Frontmatter
{
"name": "run-status-monitor",
"description": "Use when probing the status of an existing job — queued, stuck, running, or finished — across local, SLURM, RunAI, or SSH. Not for launching new jobs (use run-experiment). Not for debugging NaN\/OOM\/engineering failures (use experiment-debugger). Not for interpreting valid but surprising results (use result-diagnosis).",
"allowed-tools": "Read, Write, Edit, Bash, Glob",
"argument-hint": "[run-id] [--config .agent\/run-status\/runs.yaml] [--output docs\/ops\/runs\/<run-id>-status.md]"
}
Run Status Monitor
Answer lightweight operational questions about active runs while keeping raw logs out of the main conversation.
Use this skill for:
- "现在实验进行到哪里了？"
- "有什么中间结果？"
- "预计还要多久结束？"
- "这个 job 是卡住了还是还在跑？"
Do not paste long scheduler output or training logs into chat. Probe, compress, write a short status artifact, and report only the summary.
Skill Directory Layout
<installed-skill-dir>/
├── SKILL.md
├── scripts/
│ └── run_status_probe.py
├── references/
│ └── backends.md
└── templates/
├── runs.yaml
└── status.md
Core Contract
- Raw logs stay in their original location or private
.agent/run artifacts. - The main agent reads only a short generated status artifact.
- Do not run open-ended
sleep/poll/log-watch loops in the main agent transcript. A single bounded probe is acceptable; repeated checks must be done through a status artifact, a project wrapper, or a sidecar/background monitor that writes a short artifact. - Prefer project/private wrapper commands for server-specific probes. For SSH-backed status checks, prefer
remote-cmdfor simple commands andremote-bashfor project scripts or any command containing loops,$variables, command substitution, pipes, globs,find, orawk. - Use
sidecar-task-runneror a project-local monitor artifact when status tracking needs more than one check, noisy log interpretation, multiple jobs, or delayed follow-up. The sidecar/monitor should own polling and log compression; the main agent should read only its final short artifact. - Use an authentication circuit breaker for scheduler/API probes. If a RunAI/Kubernetes/cluster API command reports OAuth/session refresh failure such as
invalid_grant, stop retrying API probes in this turn, mark API monitoring blocked, and switch to filesystem/project-wrapper fallback when available. - If a run appears failed, stale, or scientifically surprising, route to
result-diagnosisafter creating the status artifact. - If a run is pending, distinguish scheduler/resource causes from code causes. Summarize whether the blocker appears to be pool/partition capacity, quota/fair-share, CPU/memory request, image pull,
ContainerCreating, environment startup, or unknown, and recommend the smallest compatible next action. - If a run is spending time creating or syncing a new uv environment, report that as environment setup overhead. Check whether the job used an existing project/stage env or created a job-specific env, and flag avoidable env proliferation.
- Report resource occupancy when available: allocated GPU count, active GPU count, per-GPU utilization/memory, node/process ownership, and whether the observed usage matches the workload shape.
- If a job is running normally but leaves allocated GPUs idle, mark it
underutilizedrather than simply healthy, and recommend the next launch shape: fewer GPUs, scheduler array, per-GPU worker pool, or native multi-GPU launcher.
Expected Project Layout
.agent/run-status/
├── runs.yaml # project-local run monitor config, private if it contains hosts/paths
└── raw/ # optional raw probe captures, ignored/private
docs/ops/runs/
└── <run-id>-status.md # short status artifact safe for main-agent reading
Workflow
- Locate project root.
- Read
references/backends.md. - Locate run config in this order:
- user-provided
--config .agent/run-status/runs.yamldocs/ops/runs.yamlinfra/remote-projects.yamlplus project-specific notes
- user-provided
- Run the probe script when a config exists:
python3 <installed-skill-dir>/scripts/run_status_probe.py \
--config .agent/run-status/runs.yaml \
--run <run-id>
- If no config exists, create one from
templates/runs.yamlwith the minimum backend fields and ask only for missing run identity fields. - Read only the generated
status_artifact, not raw logs, before answering the user. - Update
docs/ops/current-status.mdor project memory only when the status changes durable project state — durable state includes: run completed with surprising or paper-facing metrics, confirmed failure with identified cause, resource occupancy pattern that should inform the next launch policy, or a new run ID that becomes the canonical reference for a claim or experiment.
For ongoing monitoring:
- Write a small monitor plan under
.agent/run-status/or use a project wrapper that writesdocs/ops/runs/<run-id>-status.md. - If waiting is needed, keep it outside the main transcript as a bounded background/session task and have it overwrite the short status artifact.
- Return to the main agent with only the status artifact path and its compressed fields.
- Stop monitoring when the run reaches a terminal state, auth is blocked, or the next useful check is far enough away that a human reminder is cheaper than agent polling.
Output Rules
Every user-facing answer should fit this shape:
Run: <id>
State: running | pending | succeeded | failed | stale | unknown
Progress: <short>
Resources: <allocated vs active GPUs, utilization, memory, or unknown>
Latest metrics: <short>
Last update: <time or unknown>
ETA: <estimate or unknown>
Risk: <short>
Artifact: <status artifact path>
Escalate when:
- the run is failed/stale
- the run is pending and a cheaper or lower-wait compatible resource could unblock a smoke/debug check
- the run is stuck in image pull or
ContainerCreatinglong enough to consume the smoke/debug budget - metrics are surprising
- logs show repeated exceptions, OOM, NaN, or checkpoint failures
- scheduler/API auth is blocked and needs a single explicit login refresh action
- ETA cannot be estimated because progress markers are absent
- allocated resources are idle or only partly used and the workload could be packed or sharded more effectively
- the probe command needs network/server approval
Version History
- e0220ca Current 2026-07-05 21:37
