TensorPool GPU Cloud

On-demand GPU clusters and git-style training jobs via the tp CLI. TensorPool provides multi-node GPU clusters with high-speed interconnects, persistent storage, and SLURM for distributed training.

When to Use TensorPool

Use TensorPool when:

• Need on-demand GPU clusters with SSH access (single or multi-node)
• Running distributed training across multiple nodes (SLURM pre-installed)
• Want git-style job interface: tp job push to submit, tp job pull to get results
• Need persistent NFS storage shared across cluster nodes
• Require B200, B300, H200, H100, L40S, or MI300X GPUs
• Want pay-per-second billing with no egress fees

Key features:

• GPU variety: B300, B200, H200, H100, L40S, MI300X, CPU instances
• Multi-node clusters: 8xB200 and 8xH200 with SLURM + InfiniBand
• Jobs: Git-style tp job push/pull/listen for batch experiments
• Persistent storage: Shared NFS volumes (300 GB/s aggregate) or S3-compatible object storage
• Simple pricing: Per-second billing, H100 at $1.99/hr, H200 at $2.99/hr, B200 at $4.99/hr

Use alternatives instead:

• Tinker: For managed SFT/fine-tuning (no infrastructure management)
• Prime Intellect Lab: For hosted RL training with environments
• Modal: For serverless, auto-scaling GPU workloads
• Lambda Labs: For dedicated instances with persistent filesystems
• SkyPilot: For multi-cloud orchestration and cost optimization

Decision Matrix

Credential Setup

Credentials are auto-injected by openscience when connected via the dashboard.

# Verify credentials
[ -n "$TENSORPOOL_KEY" ] && echo "TENSORPOOL_KEY set" || echo "NOT SET"

If not set: connect TensorPool at https://app.syntheticsciences.ai -> Services, then restart openscience.

Quick Start

Installation

pip install tensorpool

Authentication

# Set API key (synced automatically via OpenScience dashboard)
export TENSORPOOL_KEY="your_api_key_here"

# Verify
[ -n "$TENSORPOOL_KEY" ] && echo "set" || echo "not set"

If connected via the Synthetic Sciences dashboard, TENSORPOOL_KEY is injected automatically.

Create Your First Cluster

# Single H100
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xH100

# Check status
tp cluster info <cluster_id>

# SSH in
tp ssh <instance_id>

# Destroy when done
tp cluster destroy <cluster_id>

Submit a Training Job

# Initialize job config
tp job init

# Edit tp.config.toml, then push
tp job push tp.config.toml

# Stream logs
tp job listen <job_id>

# Download results
tp job pull <job_id>

Instance Types

Pricing (per GPU/hour)

All charges prorated to the second.

Clusters

Single-Node Clusters

# Various GPU configs
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xH100
tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xH200
tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xB200
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xL40S

# With custom name
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xH100 --name my-cluster

Multi-Node Clusters

Multi-node clusters come with SLURM preinstalled. Only 8xH200 and 8xB200 support multi-node.

# 2-node cluster (16 GPUs total)
tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xH200 -n 2

# 4-node cluster (32 GPUs total)
tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xB200 -n 4

Multi-node architecture:

• Jumphost: {cluster_id}-jumphost — SLURM login/controller, public IP
• Worker nodes: {cluster_id}-0, {cluster_id}-1, etc. — private IPs only

# SSH into jumphost first
tp ssh <jumphost-instance-id>

# From jumphost, access workers
ssh <cluster_id>-0
ssh <cluster_id>-1

Cluster Management

tp cluster list                    # List all clusters
tp cluster list --org              # List organization clusters
tp cluster info <cluster_id>       # Detailed cluster info
tp cluster edit <cluster_id> --name "new-name"
tp cluster edit <cluster_id> --deletion-protection true
tp cluster destroy <cluster_id>    # Terminate cluster

Cluster Statuses

PENDING → PROVISIONING → CONFIGURING → RUNNING → DESTROYING → DESTROYED

If any instance fails, cluster shows as FAILED.

Jobs

Git-style interface for running training experiments on GPUs. Pay only for the time your job runs.

Job Configuration (tp.config.toml)

commands = [
    "pip install -r requirements.txt",
    "python train.py --epochs 100",
]

instance_type = "1xH100"

outputs = [
    "checkpoints/",
    "model.pth",
    "results.json",
]

ignore = [
    ".venv",
    "venv/",
    "__pycache__/",
    ".git",
    "*.pyc",
]

Job Commands

tp job init                        # Create tp.config.toml
tp job push tp.config.toml         # Submit job
tp job list                        # List your jobs
tp job list --org                  # List org jobs
tp job info <job_id>               # Job details
tp job listen <job_id>             # Stream real-time logs
tp job pull <job_id>               # Download output files
tp job pull <job_id> --force       # Overwrite existing files
tp job cancel <job_id>             # Cancel running job
tp job cancel <job_id> --no-input  # Skip confirmation

Job Statuses

Pending → Running → Completed / Error / Failed / Canceled

• Error: User-level (non-zero exit code) — check logs
• Failed: System-level (node/GPU failure) — TensorPool investigates

Multiple Experiments

# Create multiple configs
tp job init  # → tp.config.toml (rename to tp.baseline.toml)
tp job init  # → tp.config1.toml (rename to tp.experiment.toml)

# Run different experiments
tp job push tp.baseline.toml
tp job push tp.experiment.toml

Storage

Shared Storage Volumes (NFS)

High-performance NFS for multi-node clusters. Up to 300 GB/s aggregate read throughput.

# Create 500GB shared volume
tp storage create -t shared -s 500 --name training-data

# Attach to cluster
tp cluster attach <cluster_id> <storage_id>

# Access on cluster at /mnt/shared-<storage_id>

# Detach
tp cluster detach <cluster_id> <storage_id>

# Destroy
tp storage destroy <storage_id>

Shared storage: Multi-node only (2+ nodes), $100/TB/month, POSIX compliant.

Object Storage (S3-compatible)

# Create object storage bucket
tp storage create -t object --name models

# Attach to any cluster type
tp cluster attach <cluster_id> <storage_id>

# Mount at /mnt/object-<storage_id> (FUSE)
# Prefer boto3/rclone over FUSE mount for performance

Object storage: All cluster types, $20/TB/month, globally replicated, no ingress/egress fees. Not POSIX compliant.

Storage Commands

tp storage create -t <type> [-s <size>] [--name <name>]
tp storage list
tp storage info <storage_id>
tp storage edit <storage_id> --name "new-name"
tp storage edit <storage_id> --deletion-protection true
tp storage destroy <storage_id>

SSH Keys

# Generate if needed
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519

# Use when creating clusters
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xH100

# Connect to cluster
tp ssh <instance_id>

Common Workflows

Workflow 1: Single-Node Training Job

# 1. Create job config
tp job init

# 2. Configure tp.config.toml
# commands = ["pip install -r requirements.txt", "python train.py"]
# instance_type = "1xH100"
# outputs = ["checkpoints/", "model.pth"]

# 3. Submit
tp job push tp.config.toml

# 4. Monitor
tp job listen <job_id>

# 5. Get results
tp job pull <job_id>

Workflow 2: Multi-Node Distributed Training

# 1. Create 4-node cluster with storage
tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xH200 -n 4
tp storage create -t shared -s 1000 --name dataset
tp cluster attach <cluster_id> <storage_id>

# 2. SSH into jumphost
tp ssh <jumphost-instance-id>

# 3. Upload data to shared storage
cd /mnt/shared-<storage_id>
# rsync, wget, or HF download your dataset here

# 4. Submit SLURM job
srun --nodes=4 --ntasks-per-node=8 --gpus-per-node=8 \
  torchrun --nnodes=4 --nproc_per_node=8 \
  --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29500 \
  train.py

# 5. Clean up
tp cluster detach <cluster_id> <storage_id>
tp cluster destroy <cluster_id>

Workflow 3: Interactive Development

# 1. Create single-node cluster
tp cluster create -i ~/.ssh/id_ed25519.pub -t 1xH100 --name dev-box

# 2. SSH in and iterate
tp ssh <instance_id>
git clone <repo>
pip install -r requirements.txt
python train.py

# 3. Destroy when done
tp cluster destroy <cluster_id>

Troubleshooting

Common Issues

1. TENSORPOOL_KEY not set

[ -n "$TENSORPOOL_KEY" ] && echo "set" || echo "not set"
# If not set, connect via OpenScience dashboard or export manually

2. Cluster stuck in PENDING/PROVISIONING

# Check cluster status
tp cluster info <cluster_id>
# Try a different instance type or wait for capacity

3. Can't SSH into cluster

• Wait for status to reach RUNNING (can take a few minutes)
• Verify SSH key was provided at cluster creation
• For multi-node: SSH into jumphost first, then access workers

4. Multi-node workers not accessible

# Workers have private IPs only — must go through jumphost
tp ssh <jumphost-instance-id>
ssh <cluster_id>-0  # from jumphost

5. Storage attachment fails

• Shared storage: only multi-node clusters (2+ nodes)
• Object storage: works on all cluster types
• Check storage status is READY before attaching

6. Job stuck in Pending

tp job info <job_id>
# Check instance type availability
tp job cancel <job_id>  # Cancel and retry if needed

7. Job Error (non-zero exit code)

# Stream logs to see what failed
tp job listen <job_id>
# Fix script, re-push
tp job push tp.config.toml

8. Object storage slow for small files

• Object storage has per-request overhead (HTTP calls)
• Use boto3 or rclone instead of FUSE mount
• Don't set up Python venvs on object storage (thousands of small files)

Agent Usage Instructions

When the openscience agent loads this skill for a user task:

1. Check credentials first: Verify TENSORPOOL_KEY is set
2. Determine cluster vs job: Jobs for batch experiments, clusters for interactive work
3. Select instance type: Match GPU to workload (H100 for training, L40S for inference, B200 for largest models)
4. ALWAYS get user approval before creating resources: Present instance type, estimated cost/hour, and expected duration. TensorPool bills per-second — user manages their own billing. Wait for explicit approval.
5. For jobs: Create tp.config.toml, show it to user, get approval, then tp job push
6. For clusters: Show the tp cluster create command with instance type and cost, get approval first
7. Monitor: Use tp job listen or tp ssh to track progress
8. Clean up: Always destroy clusters and detach storage when done

Cost Awareness

TensorPool charges per GPU/hour, prorated to the second:

• B200: $4.99/GPU/hr → 8xB200 = ~$40/hr per node
• H200: $2.99/GPU/hr → 8xH200 = ~$24/hr per node
• H100: $1.99/GPU/hr → 8xH100 = ~$16/hr per node
• L40S: $1.49/GPU/hr
• Storage: Shared $100/TB/month, Object $20/TB/month

ALWAYS present estimated cost before creating any resource.

Example Agent Workflow

User: "Set up a 2-node H200 cluster for distributed training"

Agent steps:
1. Load skill: tensorpool-gpu-cloud
2. Check TENSORPOOL_KEY is set
3. Present cost estimate: 2x 8xH200 = $47.84/hr ($0.80/min)
4. Wait for explicit user approval
5. tp cluster create -i ~/.ssh/id_ed25519.pub -t 8xH200 -n 2
6. Wait for RUNNING status
7. tp ssh <jumphost-instance-id>
8. Help user with training setup
9. Remind user to destroy cluster when done

Quick Reference

Resources

• Documentation: https://docs.tensorpool.dev
• Dashboard: https://dashboard.tensorpool.dev
• Pricing: https://tensorpool.dev/pricing
• Community: https://tensorpool.dev/slack
• Support: team@tensorpool.dev