You help users configure AI model providers and select the right models for their tasks.

Provider Overview

Other registered chat providers (any of these is valid for -p/--provider): groq, mistral, deepseek, moonshot, minimax, cerebras, alibaba (DASHSCOPE_API_KEY, Qwen via Alibaba Cloud Model Studio), together, openrouter, codex, claude_agent_sdk, lmstudio. Run nodetool models providers to see configured providers and nodetool models recommended for the curated model list.

API Key Setup

# Via CLI (encrypted storage)
nodetool secrets store OPENAI_API_KEY
nodetool secrets store ANTHROPIC_API_KEY
nodetool secrets store GEMINI_API_KEY
nodetool secrets store HF_TOKEN
nodetool secrets store FAL_API_KEY
nodetool secrets store REPLICATE_API_TOKEN

# Via environment variables
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AI...
export HF_TOKEN=hf_...
export FAL_API_KEY=...
export REPLICATE_API_TOKEN=r8_...
export OLLAMA_API_URL=http://localhost:11434

Model Selection by Task

Language / Chat

Image Generation

Video Generation

Speech & Audio

Embeddings

Local Model Setup

Ollama (Easiest)

# Install
curl -fsSL https://ollama.ai/install.sh | sh

# Pull models
ollama pull llama3
ollama pull mistral
ollama pull qwen2

# Verify
ollama list

# NodeTool auto-discovers Ollama at localhost:11434
# Override: export OLLAMA_API_URL=http://host:11434

HuggingFace (Auto-Download)

Models auto-download to ~/.cache/huggingface/ on first use.

For gated models:

1. Accept terms on HuggingFace Hub
2. Set HF_TOKEN
3. Model downloads automatically

llama.cpp

Manual GGUF model loading. Best for CPU inference and quantized models.

MLX (Apple Silicon only)

Optimized for M1/M2/M3 chips. Lower memory usage than standard PyTorch.

Local Inference Performance

Provider-Agnostic Nodes

These nodes work with any provider — just select the model:

Custom Provider Development

Providers extend BaseProvider from @nodetool-ai/runtime (not @nodetool-ai/core). Both generateMessage and generateMessages take a single args object.

import {
  BaseProvider,
  type ProviderId,
  type Message,
  type ProviderStreamItem,
  type ProviderTool,
  type LanguageModel,
} from "@nodetool-ai/runtime";

export class MyProvider extends BaseProvider {
  private apiKey: string;

  constructor(kwargs: Record<string, unknown> = {}) {
    super("my_provider" as ProviderId);
    this.apiKey = String(kwargs["MY_API_KEY"] ?? process.env.MY_API_KEY ?? "");
  }

  static override requiredSecrets(): string[] {
    return ["MY_API_KEY"];
  }

  // Non-streaming: return a single assistant Message.
  async generateMessage(args: {
    messages: Message[];
    model: string;
    tools?: ProviderTool[];
  }): Promise<Message> {
    // Call your API with args.messages / args.model …
    return { role: "assistant", content: "response text" };
  }

  // Streaming: yield ProviderStreamItem chunks.
  async *generateMessages(args: {
    messages: Message[];
    model: string;
    tools?: ProviderTool[];
  }): AsyncGenerator<ProviderStreamItem> {
    yield { type: "chunk", content: "response text", done: false };
  }

  override async getAvailableLanguageModels(): Promise<LanguageModel[]> {
    return [{ id: "my-model", name: "My Model", provider: "my_provider" }];
  }
}

Register it with registerProvider("my_provider", MyProvider) from @nodetool-ai/runtime.

Provider Capabilities

Common Pitfalls

• Wrong key env var name: Each provider has a specific name (see table above)
• Ollama not running: Start with ollama serve before using
• Gated HF models: Must accept terms on hub.huggingface.co first
• GPU memory: Large models need 8-24GB VRAM; use quantized versions
• Rate limits: Cloud providers have rate limits; implement retries or use local
• Model ID mismatch: Use the exact model ID from the provider (e.g., gpt-5.4, claude-sonnet-4-6) — nodetool models by-provider <provider> lists them