remote-compute-ssh

This skill covers remote compute over SSH, including direct execution and Slurm submission: listing hosts, creating handles, running short remote commands (callCommand), reading/writing host knowledge docs, and the full async job lifecycle — submit → save job_id → read non-blocking snapshots by that ID → harvest → analysis turn → publish artifacts.

Where host.compute runs: host.compute lives ONLY on the control-plane REPL kernel — run every example below with the repl_execute tool (JavaScript), the same kernel that hosts host.mcp. The python/r data kernels have NO host.compute (SSH and approvals stay outside the sandbox workspace); calling it from a python/r cell will fail with host.compute is undefined.

Choose an execution location

Only Compute Hosts enabled for this Session are visible or callable. Discover them in one catalog; each entry has role selected or available. A non-empty selected pool is an execution instruction: run tool-backed task work on one or more selected hosts as the task requires. The pool has no priority and does not imply automatic multi-host scheduling. If no host is selected, choose from the available entries. Read details() only for candidates that need closer evaluation.

Never guess or reuse a provider id absent from the catalog. A user naming a disabled host does not make it callable; explain that it must first be enabled for this Session. If no eligible host is usable, explain the blocker and ask the user how to proceed.

const hosts = await host.compute.listHosts()
const selectedHosts = hosts.filter((host) => host.role === 'selected')
const candidates = selectedHosts.length > 0 ? selectedHosts : hosts

Each list item is a compact summary with provider_id, display_name, shape, execution_mode, status, and role (last_probe_ok, probe_failed, or not_probed). last_probe_ok means the most recent persisted Probe succeeded; it does not assert live connectivity. Knowledge documents and resource probe snapshots are deliberately excluded from discovery results.

API reference

// List this Session's enabled hosts as one role-bearing compact catalog
const hosts = await host.compute.listHosts()

// Compatibility discovery names remain available; both still hide disabled hosts.
const visibleHosts = await host.compute.listRegistered()
const selectedHosts = await host.compute.listPreferred()

// Create a handle to a specific host (no network call)
const c = host.compute.create('ssh:<alias>')

// Run a short remote command (throws on approval_denied / host_unreachable / timeout)
const result = await c.callCommand('<shell command>', '<one-line intent for the approval card>', {
  loginShell: true, // default: true — runs login profiles, then readable ~/.bashrc, before this command
  timeoutSeconds: 60 // optional — the host applies its own default (60s) when omitted
})
// result → { exit_code, stdout, stderr, truncated }

// Read the persisted operation instructions and the independent resource probe snapshot.
// doc is always the exact saved text (including '' before instructions are saved).
// probe is explicitly null when this host has never been probed.
const info = await host.compute.details('ssh:<alias>', { mode: 'read' })

// Append a note to the persisted host knowledge doc (agent writes; 32 KB cap enforced).
// Append changes only doc; it never copies or changes probe observations.
await host.compute.details('ssh:<alias>', {
  mode: 'append',
  text: '\n## Note\nlearned X on <date>'
})

// Alternatively, replace the entire host knowledge doc. Read again before replacing,
// especially if you appended above: oldText must match the persisted doc exactly.
const latest = await host.compute.details('ssh:<alias>', { mode: 'read' })
await host.compute.details('ssh:<alias>', {
  mode: 'replace',
  text: '<new full doc>',
  oldText: latest.doc
})

Treat doc as operation instructions and durable host knowledge. Treat probe as a dated observation: resource values may change, and detecting a scheduler does not authorize submitting a job or select an account, partition, or queue. On a document mismatch or details_conflict error, read again and merge your draft with the latest document before retrying. A resource-only probe refresh does not change doc or cause a replacement conflict. After writing, read again to verify the saved contents. A successful append or replace result is only { ok: true }. The write result does not contain doc or probe; always read again to verify the exact persisted doc rather than reading fields from the write result.

With loginShell: true, the remote Bash login profiles run first and then Open Science attempts to source ~/.bashrc when it is readable. A .bashrc can deliberately return early for non-interactive shells, so variables declared after such a guard are not available. A missing .bashrc is a no-op. Set loginShell: false to run the command without either initialization step. Initialization failures are reported through the normal command result/error behavior.

API reference (async jobs)

Use submitJob for long-running computations (minutes to hours). It returns immediately with a job_id; the job runs on the remote host in the background. When the job finishes, the app automatically harvests the outputs and initiates a new analysis turn if you have not already read the terminal result. Save the exact job_id from the submission result in your working context; there is intentionally no historical Job scan for rediscovering it. Status and result reads use only that saved ID and return non-blocking local snapshots.

For a local input, src is relative to the Agent Session workspace—the same workspace used by file writing tools. Write a script or small generated input there, then pass its relative path. Open Science snapshots accepted inputs before approval and dispatch. Do not pass arbitrary absolute local paths or copy files into app-managed notebooks/... directories. An absolute src is valid only when it is the exact path returned by host.artifactPath(versionId) or an exact registered Session input path already supplied in the Notebook context.

// Reuse the `candidates` selected above from the Session catalog.

// Submit a non-blocking job — returns immediately after the user approves.
// The Compute Host's configured execution mode selects direct SSH or Slurm.
const c = host.compute.create('ssh:<alias>')
const job = await c.submitJob(
  '<one-line intent for the approval card>', // shown in the approval card
  '<shell command>', // command to run remotely
  {
    environment: 'protein-gpu', // optional logical name; see Environment activation below
    timeoutSeconds: 3600, // optional; default 24 h, max 7 days
    inputs: [
      { src: 'in.dat', dstFilename: 'in.dat' }, // stage an Agent Session workspace file
      { remotePath: 'ssh:<alias>/<abs_path>' } // link a remote file (no transfer)
    ],
    outputs: [
      '*.result', // featured (default visibility)
      { glob: '*.json', visibility: 'featured' }, // explicitly featured
      { glob: '*.log', visibility: 'hidden' }, // hidden (diagnostic, not shown in card)
      { glob: 'checkpoints/**', residency: 'remote' } // leave on remote — recorded in left_on_remote
    ],
    harvest: {
      exclude: ['work/**'], // never harvest these paths
      maxFileMb: 100, // single-file hard maximum (100 MiB)
      maxTotalMb: 500 // per-job hard maximum, including stdout/stderr (500 MiB)
    }
  }
)
// job → { job_id, provider_id, status: 'submitted' | 'queued', remote_workdir }
const savedJobId = job.job_id // retain this exact id for the later dependent step
return { ...job, job_id: savedJobId }

Read a saved Job snapshot

Use the saved ID when the Job's state or result is relevant. .status() and .result() are non-blocking local reads in every state; neither waits for completion, triggers SSH, or starts another harvest. .result() also includes harvested file lists. Both calls report follow_up_delivery. A final .result() read returns suppressed when it prevents the fallback, or committed when that fallback already crossed its dispatch fence. A .status() snapshot remains pending because it omits harvested file lists. Use the submission's exact ID rather than searching old Jobs.

const snapshot = await c.attachJob(savedJobId).result()
if (!snapshot.result_final) {
  return {
    job_id: savedJobId,
    status: snapshot.status,
    result_final: false,
    follow_up_delivery: snapshot.follow_up_delivery
  }
}
return snapshot

Treat only result_final: true as the final result; a provider-terminal status can still be waiting for local harvest. The app owns provider polling and harvest in the background. An unread final result is delivered in a later Agent Turn. A final .result() snapshot reports follow_up_delivery: 'suppressed' when it suppresses that fallback, or committed if automatic delivery already won the race and remains authoritative. .status() never consumes the full result.

Direct SSH or Slurm

The Compute Host's configured execution mode selects how every job is launched. direct_ssh runs the command as a detached process on the SSH target. slurm submits it with sbatch; put the cluster's required #SBATCH directives at the top of command. Open Science owns submission, scheduler-status polling, cancellation, and harvest. Do not call sbatch, squeue, or scancel around submitJob yourself.

Read listHosts() for the configured mode and details() for provider-specific directives; do not try to override the mode per job or infer it only from the workload. If Slurm is unavailable or rejects the script, report the returned error and the concrete next step (for example, add an account or partition directive). Do not silently rerun the workload directly on a login node.

Open Science accepts ordinary single-job directives such as partition, account, CPUs, memory, and GPUs. Set timeoutSeconds for the workload runtime. You may set the scheduler allocation limit with one #SBATCH --time=value directive; when it is absent, Open Science derives a default allocation limit from timeoutSeconds. Open Science owns the job name, working directory, stdout, and stderr directives. Avoid job arrays because one Open Science job tracks one scheduler job and one output harvest. Submit independent work as separate jobs and use the Session concurrency limit when needed.

For Slurm, request resources with one #SBATCH --option=value directive per line (or a value-free flag such as #SBATCH --exclusive). The legacy resources option is descriptive metadata; it does not allocate CPUs, memory, or GPUs. timeoutSeconds limits workload runtime, not queue wait; #SBATCH --time sets the scheduler allocation limit. Neither is a promise of queue start time.

The non-blocking job status() and result() snapshots include scheduler_job_id when known, error_code on failure, and last_poll_error when observation or submission recovery needs attention. A pending reason or delayed accounting row does not mean the workload failed. If a submission is unconfirmed, use the reported job identity and provider diagnostics before deciding whether to submit again; Open Science does not automatically submit a duplicate.

Environment activation

The optional environment value is a logical name, not a shell command. Open Science sources ~/.openscience/environments/<name>.sh before the workload for direct and Slurm jobs. Names are 1–64 letters, numbers, periods, underscores, or hyphens and must start with a letter or number. The file and every software/cache path it references must be visible on the execution node.

If a submission reports that this activation file is missing, load the Compute Environment Setup Skill to prepare exact setup, repair, and removal instructions for the user or host administrator to run outside Open Science. Validate the user-managed activation after they apply the plan, then retry. Do not guess a conda name, add an inline install to the science job, or hide activation in .bashrc. Omit environment when the command deliberately uses the host's default environment.

Harvest safety boundaries

• Declared output files are selected before stdout and stderr; logs use the remaining per-job budget.
• The app rejects model-supplied limits above 100 MiB per file or 500 MiB per job.
• Harvest also preserves a fixed 2 GiB of free local disk space. Files that do not fit remain remote.

Behavior boundaries

• While the job runs: the conversation is open. The user can send messages; you can handle other tasks. Each status/result query returns immediately with the current local snapshot.
• When the job finishes: if you did not actively read its terminal result, the app initiates a new analysis turn automatically. You do not trigger this fallback.

Check job status (non-blocking read, for informational use)

// Non-blocking DB read — no SSH. Use if you need a status snapshot mid-conversation.
const handle = c.attachJob(job.job_id)
const s = await handle.status()
// s → {
//   job_id, scheduler_job_id?, status, result_final, cancellation_status?, exit_code,
//   error_code?, last_poll_error?, stdout_tail, stderr_tail, remote_workdir,
//   follow_up_delivery: 'pending'
// }
// status: 'queued' | 'submitted' | 'running' | 'success' | 'failed' | 'timeout' | 'error'
// result_final is the authority for whether local harvest is complete; status alone is not.

To stop one active job, request durable cancellation through the same handle:

await c.attachJob(job.job_id).cancel()
// cancellation_status is 'cancelling' until owned remote termination is confirmed,
// then 'cancelled'. Repeating cancel() is safe.

submitJob status values

Workflow: the analysis turn

When the app initiates the analysis turn, it provides the job_id, status, and featured_files (Notebook Session-relative paths under hpc/<job_id>/featured/). In this turn:

1. Call attachJob(job_id).result() to get the full result dict.
2. Inspect the outputs, run any analysis needed.
3. Call write_artifact_file to publish outputs worth keeping as artifacts.

// In the analysis turn — read the full harvested result (non-blocking DB + directory scan)
const c = host.compute.create('ssh:<alias>')
const r = await c.attachJob(job_id).result()
// r → {
//   job_id, status, result_final, exit_code,
//   local_output_root: '/absolute/path/to/this/notebook/session',
//   producer_run_id: 'notebook-run-...',
//   featured_files: ['hpc/<job_id>/featured/out.result', ...],   // Notebook Session-relative
//   hidden_files:   ['hpc/<job_id>/hidden/run.log', ...],
//   output_files:   [...featured_files, ...hidden_files],         // featured first
//   left_on_remote: [{ uri: 'ssh:<alias>/<abs_path>', size_mb: 420, reason: 'residency:remote' }],
//   remote_workdir: '.openscience/jobs/<job_id>',
//   stdout_tail: '...last 64 KB...',
//   stderr_tail: '...last 64 KB...'
// }

Harvested files use hpc/<job_id>/ paths inside the Notebook Session, relative to r.local_output_root, its absolute root. This is separate from the Agent Session workspace used to resolve a submitted relative src. In the automatic analysis turn, join the returned root and relative output path; do not copy files between app-managed directories. For example:

# Substitute the exact root and featured path returned by result().
from pathlib import Path
import pandas as pd
df = pd.read_csv(Path('<local_output_root>') / 'hpc/<job_id>/featured/results.csv')

Publish artifacts

Harvest only lands files in the Notebook Session — it does NOT publish artifacts automatically. Call the write_artifact_file tool exposed by the open-science-artifacts server directly in the analysis turn, outside repl_execute. Do not call it through host.mcp or guess a Connector alias. Pass an absolute source.path formed by joining r.local_output_root with the corresponding entry in r.featured_files:

{
  "filename": "results.csv",
  "mimeType": "text/csv",
  "producerRunId": "<producer_run_id>",
  "source": {
    "kind": "localPath",
    "path": "<local_output_root>/hpc/<job_id>/featured/results.csv"
  }
}

Repeat the direct tool call for each output in r.featured_files worth publishing, mapping each path the same way. Pass r.producer_run_id as the top-level producerRunId; it identifies the Notebook submission run that owns the Compute Job and lets the artifact retain that execution lineage across analysis turns. Do not substitute the current analysis run id or guess an id. Artifacts appear in the artifact panel with provenance tied to the compute execution and this analysis turn.

When the job fails

Read r.exit_code and r.stderr_tail. An infrastructure failure (wrong partition, env not activated, missing module, OOM, walltime) is yours to fix — adjust command, record the fix, fresh c.submitJob(). A harvest failure (r.stderr_tail notes it, r.remote_workdir is preserved) means some files were not downloaded — the remote workdir is kept so you can c.callCommand('ls ...', intent='...') to inspect what's there.

Chaining jobs via left_on_remote

Large outputs declared with residency: 'remote' or files that exceed the size threshold stay on the remote host and appear in r.left_on_remote. Use their URIs directly as remotePath inputs to the next job — no local round-trip:

// In the analysis turn — chain a left_on_remote output into the next job
const big_output_uri = r.left_on_remote[0].uri // e.g. 'ssh:biowulf//scratch/jobs/<id>/big.h5'

const job2 = await c.submitJob(
  'process big.h5 output from job 1',
  'python process.py --input big.h5 --out summary.csv',
  {
    inputs: [
      { remotePath: big_output_uri } // symlinked in job workdir, no transfer
    ],
    outputs: ['summary.csv']
  }
)

Submitting several jobs

Submit a batch and let each job's analysis turn handle its results independently. The app triggers a separate analysis turn for each job as it finishes (or merges simultaneous completions into one turn with multiple job_ids):

// Submit multiple jobs — end the cell after all submits
const c = host.compute.create('ssh:gpu-cluster')
const jobs = []
for (const seed of [0, 1, 2, 3, 4]) {
  const job = await c.submitJob(
    `AlphaFold seed ${seed}`,
    `python fold.py --seed ${seed} --in input.fasta --out ranked.pdb`,
    {
      inputs: [{ src: 'input.fasta', dstFilename: 'input.fasta' }],
      outputs: [{ glob: '*.pdb', visibility: 'featured' }],
      timeoutSeconds: 3600
    }
  )
  jobs.push(job.job_id)
}
return jobs // preserve every exact ID for later status/result reads

The app may trigger an analysis turn for each unread completion (or a merged turn for simultaneous completions). A final result read reports whether that Job's follow-up was suppressed or committed.

Session concurrency control

Cap how many non-terminal jobs run at once across all providers in this conversation. Jobs that would exceed the cap enter a queued state and auto-dispatch when a slot frees up. These two methods live on the handle returned by create(), but they are session-scoped — they act on the whole conversation, not on the handle's bound provider.

const c = host.compute.create('ssh:<alias>')

// Set the conversation-wide limit (positive integer 1..500).
await c.setConcurrencyLimit(2)

// Read the session's concurrency status (non-blocking DB read, no SSH).
const s = await c.status()
// s → {
//   session_limit: number | null,            // the cap you set, or null if unset
//   active_count: number,                    // non-terminal jobs running now
//   queued_count: number,                    // jobs waiting for a slot
//   provider_ceilings: Record<string, number> // per-host hard limits (host config)
// }

callCommand error handling

try {
  const r = await c.callCommand('cmd', '<intent>')
} catch (e) {
  const code = e.error_code || ''
  if (code === 'host_unreachable') {
    // SSH connectivity issue — needs user action (VPN, key, etc.); e.retry_after_user_action is true
  } else if (code === 'approval_denied') {
    // User declined the approval card
  } else if (code === 'timeout') {
    // Command exceeded timeoutSeconds
  }
}

Typical first-contact workflow

1. await host.compute.details(provider_id, { mode: 'read' }) — read saved operation instructions from doc and inspect the separate, dated probe observation. An empty doc means no instructions have been saved; it says nothing about whether the host has been probed.
2. Bind once: const c = host.compute.create(provider_id).
3. Run one batched probe: await c.callCommand('id; module avail 2>&1 | head -40', '<intent>').
4. Append what you learned via await host.compute.details(..., { mode: 'append' }).

What to record in the knowledge doc

The knowledge doc is the only state that survives across sessions. Record:

• Scheduler type and any known partition/account combinations that worked.
• Environment activation commands (e.g. module load X/<ver>, conda activate <env>).
• Verified invocations tagged verified <date>; user-provided info tagged per user <date>.
• Gotchas specific to this host or provider.

Do NOT record per-job state, transient errors, or facts about your project — those belong elsewhere. When a session ends without new host-specific learnings, write nothing.