Monitor a run

Runbook

On launch

1. Find the run dir and read the resolved configs at {run_dir}/configs/ (start with rl.json). The run dir is {output_dir}/{run_name} — run.name auto-generates as <envs>--<model>--<short-id>, so if you only know the output dir, pick the most recently modified subdirectory (ls -t {output_dir} | head -1) or read run.name from the launch command.
2. Confirm all processes are alive and the run is making progress.
3. Write the initial summary into {run_dir}/STATUS.md.

Recurring check-ins

Default cadence: 1 hour (researcher can override). At each check-in:

1. Confirm processes are alive.
2. Grep logs for errors/warnings; note current step and key metrics.
3. Append an entry to {run_dir}/STATUS.md (never overwrite):

## YYYY-MM-DD HH:MM UTC

**Step**: {current_step} / {max_steps}
**Health**: {Healthy | Degraded | Down}

**Progress**: reward/mean, seq_len, truncation, eval scores, env-specific metrics.
**Stability**: entropy, mismatch_kl, grad_norm — flag spikes.
**Performance**: trainer vs orchestrator step time, env lag, inference pressure.

**Notes**: anything unusual (errors, restarts, hangs). Omit if nothing notable.

In W&B, each project auto-gets an "overview" saved view (train / eval / stability / performance sections) on its first run — use it for a quick check instead of the auto-generated default workspace.

Restarting a run

Never restart unless the researcher explicitly asked. Confirm the exact restart command and the conditions that warrant one.

Never run kill or launch commands from your own shell. Dispatch them to the tmux Launcher window so the researcher sees what was executed:

SESSION=$(tmux display-message -p '#S')
tmux send-keys -t "$SESSION:Launcher" 'your command here' Enter

After a restart, verify all processes are back up and progress resumed before the next check-in.

Reference

Where to find things

• scripts/tmux.sh launches the run with a Launcher window in the named tmux session. The Claude window receives the run dir and session name in its appended prompt — if either is missing, ask rather than guess.
• {run_dir}/configs/ — resolved configs, written as JSON so explicit None settings round-trip (rl.json has the full picture).
• {run_dir}/logs/latest/ — the current attempt's logs (each launch gets logs/attempt_<n>/; resumes never overwrite earlier attempts). See below.
• {run_dir}/rollouts/step_N/{train,eval}/ — saved rollout traces (see Traces below).

Logs

{run_dir}/logs/latest/
├── trainer.log                # rank 0 stdout
├── orchestrator.log           # orchestrator stdout
├── evaluator.log              # SFT online-eval evaluator stdout (single-node; the decoupled multi-node eval job logs at {run_dir}/logs/evaluator.log)
├── inference.log              # vLLM stdout
├── trainer/
│   ├── node_*.log             # per-node (multi-node only)
│   └── torchrun/              # per-rank stdout/stderr
├── inference/
│   ├── node_*.log             # per-node (multi-node only)
│   └── router.log             # the single global router (multi-node only; single-node logs it in inference.log)
└── envs/{train,eval}/{env_name}.log    # one log file per env

Usually tailing trainer.log, orchestrator.log, and inference.log is enough. Drop into per-node or per-rank logs only when debugging. All logs are loguru with HH:mm:ss LEVEL message; levels: DEBUG, INFO, SUCCESS, WARNING, ERROR.

Scan for problems:

grep -E "WARNING|ERROR" {run_dir}/logs/latest/{trainer,orchestrator,evaluator,inference}.log
grep -E "WARNING|ERROR" {run_dir}/logs/latest/envs/{train,eval}/*.log

Metrics

All metrics print to the console log (and W&B when configured).

Progress — orchestrator log. Rollout metrics mirror the episode/trace hierarchy, at two levels:

• {scope}/{subset}/<metric>/<stat> — episode-level facts only: the token/turn/branch counts, summed over an episode's traces.
• {scope}/{subset}/<agent>/<metric>/<stat> — every trace-level metric (reward, truncation, errors, timing, env metrics, curriculum admission, eval scores), keyed by agent name so seats never mix. Flat over that agent's traces: one sample is one trace, so an in-episode fan-out like n solvers contributes n samples.

scope is train/agg (all train envs) or train/<env> (eval/<env> for eval); subset is all (every rollout) or effective (admitted, clean, and trainable). Single-agent envs have one agent — usually agent — and one trace per episode, so both levels agree; multi-agent envs name each seat (proposer, solver, judge, …).

Stability — trainer log:

Performance — trainer and orchestrator step independently, so comparing step times shows who's waiting on whom.

For live vLLM stats, query Prometheus directly:

curl -s http://localhost:8100/metrics | grep -E "num_requests|gpu_cache_usage"  # engine port (8000 is the router)
# vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc (→1.0 = KV cache saturated)

Traces

{run_dir}/rollouts/step_N/{train,eval}/all/traces.jsonl        # appended per rollout as it completes
{run_dir}/rollouts/step_N/{train,eval}/effective/traces.jsonl  # written per finalized batch / eval epoch

JSONL files of vf.Trace records (training tensors excluded), one line per trace — a multi-agent env's episode contributes several lines sharing one info.episode_id. all gets every completed rollout the moment it arrives — errored, curriculum-rejected, and never-batched ones included — so it's crash-durable; effective gets the clean trainable subset that went into the step's train batch (eval: the non-errored trainable epoch cohort; multiple eval envs share the step file) — untrainable traces (a frozen judge's) appear only in all. Each record carries run ({type, id, step}; for eval, step is the trigger step), verifiers (producing build), agent (model, sampling, harness, name, trainable), ok (the success sentinel — errors alone keeps retry history even after a recovery), and runtime (config + provisioned resource id, e.g. the sandbox id), plus env_name, group_id, episode_id, and policy_version under info.

wc -l {run_dir}/rollouts/step_42/train/{all,effective}/traces.jsonl
jq '.rewards' {run_dir}/rollouts/step_42/train/effective/traces.jsonl
jq 'select(.ok | not) | {id, env: .info.env_name, runtime}' {run_dir}/rollouts/step_*/train/all/traces.jsonl

The batches consumed by the trainer are shipped over ZMQ by default, so nothing binary is written. With rollout_transport.type = "filesystem" they land at {run_dir}/rollouts/step_N/rank_<rank>.bin (one packed micro-batch file per trainer DP rank), next to the trace subtrees.

Common failure modes

A few warnings are normal. Escalate when errors are persistent, growing, or hit a large fraction of rollouts.

• Env workers: exceptions in env code, timeouts, sandbox errors, OOM kills (most common source — runs user code).
• Orchestrator: empty/errored rollout spikes, weight-broadcast failures, checkpoint errors.
• Trainer: NCCL/CUDA errors, OOM, NaN loss or gradients.
• Inference: NCCL/CUDA errors, OOM, request timeouts.

Process tree

All processes use setproctitle so they're visible in ps/htop/pstree:

PRIME-RL::Launcher
├── PRIME-RL::Inference          (vLLM server, GPU 0)
├── PRIME-RL::EnvServer          (verifiers' ZMQ env server, run in-process; one per train/eval source)
│   └── Verifiers::EnvWorker0..N
├── PRIME-RL::Orchestrator       (CPU-only; connects to each env server)
├── torchrun
│   └── PRIME-RL::Trainer        (GPU 1+)
└── tail trainer.log

For multi-node runs, trainer and inference processes are on separate nodes — use srun or ssh to inspect them.