Connect Your Agent to LangWatch Simulations

Register the user's agent as an HTTP simulation target. Scenario runs call the agent's endpoint from the LangWatch backend, one HTTP request per conversation turn, and the judge verifies behavior against the traces the agent itself reports: tool calls, database writes, retrievals. Work through the steps in order, then report what changed and the first run's result.

Do NOT skip the trace adoption step (Step 4). Without it the judge can only grade the reply text, and criteria about tool calls or lookups come back inconclusive.

Step 1: Set up the LangWatch CLI

Use langwatch docs <path> to read documentation as Markdown. Some useful entry points:

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with langwatch --help and langwatch <subcommand> --help. List and get commands accept --format json for machine-readable output. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append .md to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run npx langwatch report --user-approved with a --title and --summary (or --session <transcript.jsonl>) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without --user-approved, and --dry-run prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with --email, so preview with --dry-run when the session touched sensitive data. npx langwatch report --help explains the options.

Projects and API keys: target a real project, not a personal one.

LangWatch has two kinds of project:

• Team / shared projects: real projects inside an organization. Evaluations, experiments, prompts, datasets, simulations and instrumentation must always target one of these.
• Personal projects: a private "My Workspace" scratch space tied to a single user. Never send a user's evaluations, experiments or production traces here: it is for personal exploration only, and you can mistake it for a real project.

And two ways to authenticate:

• A project API key in .env (LANGWATCH_API_KEY): the credential everything in these skills uses. It is scoped to one real project. This is the default; prefer it unless the user explicitly asks for something else.
• langwatch login --device (AI-tools / SSO): a personal device session for wrapping coding assistants (langwatch claude, langwatch codex, …). It is NOT for evaluations, prompts, datasets, scenarios or SDK instrumentation, and it points at a personal workspace. Do not run it to set up the work in these skills.

So for anything in these skills: make sure LANGWATCH_API_KEY for a real, shared project is in the project's .env. Check whether the variable is already set there before you ask for a new key, and let the CLI read the value: never print, copy or send it. Do NOT run langwatch login to pick a project, and never default to a personal project. If LANGWATCH_ENDPOINT is set, the user is self-hosted: use that endpoint instead of app.langwatch.ai.

Step 2: Locate the agent's HTTP endpoint

Find the HTTP endpoint that takes a user message and returns the agent's reply. Read the codebase first: identify the framework (FastAPI, Flask, Express, Hono, ...) and the file where the handler lives before changing anything.

If no such endpoint exists, add one:

• Accept a JSON body carrying the conversation messages.
• Run the agent.
• Return the reply text in a JSON field, for example {"reply": "..."}.

The endpoint does not need to know anything about LangWatch. The request body shape and the response parsing are configured on the LangWatch side in Step 6.

Step 3: Wire authentication for scenario traffic

Understand the endpoint's authentication before touching it.

• If the endpoint accepts a fixed token in a header, use that credential as-is in the registration (Step 6). Change nothing on the server.
• If the normal authentication is built for human users (sessions, cookies, OAuth redirects), add a dedicated authentication path for scenario traffic: the server reads the expected key from an environment variable such as SCENARIO_API_KEY and checks it against the Authorization: Bearer header on each request. A request carrying the valid key is accepted; every other request goes through the existing authentication unchanged. If SCENARIO_API_KEY is unset, the path is off.

NEVER weaken, bypass, or remove the existing authentication for normal traffic. The scenario path is additive, and the dedicated key is what the user revokes to close it.

Step 4: Adopt the trace context

The platform sends a W3C traceparent header on every call, one trace per conversation turn. When the server adopts it, the spans the agent produces land in that same trace, and the judge reads them before its verdict. A criterion like "the agent looked up the order before answering" then passes on evidence instead of on the reply's wording.

• If the service uses OpenTelemetry HTTP auto-instrumentation, adoption already happens. Verify it in the code and change nothing.
• Otherwise, attach the extracted context in a middleware that runs before any tracing starts. Do not extract inside the handler body: a handler decorated with @langwatch.trace() opens its root span before the body runs, so an extraction there is too late and the agent's spans land in a separate trace. The middleware placement covers every tracing style: decorators, with langwatch.trace(), autotrack, community instrumentations, and plain OpenTelemetry spans.

Python (ASGI middleware, e.g. FastAPI):

from opentelemetry import propagate
from opentelemetry.context import attach, detach

@app.middleware("http")
async def adopt_remote_trace(request, call_next):
    token = attach(propagate.extract(dict(request.headers)))
    try:
        return await call_next(request)
    finally:
        detach(token)

For Flask, attach in before_request (keep the token on g) and detach in teardown_request.

TypeScript (middleware, registered before the routes):

import { context, propagation } from "@opentelemetry/api";

app.use((req, res, next) => {
  const ctx = propagation.extract(context.active(), req.headers);
  context.with(ctx, () => next());
});

The TypeScript middleware needs an initialized OpenTelemetry runtime: a registered context manager and propagator. The LangWatch SDK's setupObservability() and the OpenTelemetry NodeSDK both register them at startup; without one of them, context.with and propagation.extract are no-ops.

Confirm the agent reports its traces to the same LangWatch project that runs the scenarios (the same LANGWATCH_API_KEY project). Traces sent to another project, or to another observability backend only, are invisible to the judge. If the service has no LangWatch tracing yet, set it up with the tracing skill; its prompt is "Instrument my code with LangWatch".

Step 5: ASK where the agent runs

Ask the user for the URL where this service is deployed, and wait for the answer. A staging deployment is the recommended target: it exercises the real system without touching production data. Any URL the LangWatch backend can reach works; an internal hostname or a firewalled service does not.

If the agent only runs on the user's machine, plan to use langwatch agent dev --port <port> at the end instead of a public URL: it opens a tunnel to the local port and points the registered agent at it for the session (Ctrl-C restores the previous URL). Register the agent in Step 6 as normal, then run langwatch agent dev --port <port> --agent <agent-id> and keep it running while suites execute.

Step 6: Register the agent and run the first scenario

First store the scenario key as a project secret, so the registration can reference it as {{ secrets.SCENARIO_API_KEY }} and the value stays encrypted at rest instead of readable in the agent's configuration. Ask the user to create it under Settings > Secrets in LangWatch, or run the command when they hand you a test-only value:

langwatch secret create SCENARIO_API_KEY --value "<key>"

Register the endpoint as an HTTP agent. Adjust bodyTemplate to the request shape the endpoint expects and outputPath to the JSONPath of the reply text in the endpoint's real response:

langwatch agent create 'My Agent' --type http --config '{
  "url": "https://staging.example.com/chat",
  "bodyTemplate": "{\"thread_id\": \"{{ threadId }}\", \"messages\": {{ messages }}}",
  "outputPath": "$.reply",
  "auth": {"type": "bearer", "token": "{{ secrets.SCENARIO_API_KEY }}"}
}'

The body template renders as a Liquid template on every turn. The URL and header values render the same variables:

Then create one scenario about something this agent really handles, pair it with the agent in a suite, and run it:

langwatch scenario create 'Order status question' \
  --situation "A customer asks about the status of a recent order" \
  --criteria "The agent looks up the order before answering,The agent gives a concrete delivery estimate"

langwatch suite create 'Smoke' --scenarios <scenario-id> --targets http:<agent-id>

langwatch suite run <suite-id> --wait

• Write the situation and criteria from the agent's real behavior in this codebase, not from the example above. Include at least one criterion about a tool call or a lookup, which the judge verifies against the traces from Step 4.
• --criteria takes one comma-separated string, so a criterion cannot contain a comma; rephrase instead.
• --targets takes http:<agent-id> where <agent-id> is the id langwatch agent create returned (also in langwatch agent list --format json). It is never a URL; the URL lives in the agent's config.
• --wait blocks until the run finishes and exits non-zero when it fails. Use it here: the report in Step 7 needs the result.

Step 7: Report the result

Report to the user:

• The simulations page URL of their LangWatch project (https://app.langwatch.ai/<project-slug>/simulations, or the LANGWATCH_ENDPOINT host when self-hosted).
• What changed in the codebase: the endpoint, the authentication path, the trace adoption.
• The result of the first run.

Report failures as they happened. If a CLI command failed or the platform was unreachable, name the step that failed and the error, and stop there. Do NOT claim a scenario or a suite ran when it did not.

A connected setup shows, on the run page: the conversation transcript with the reply text outputPath extracted, a trace link on each turn opening the agent's own spans, and judge reasoning that cites spans. A trace-dependent criterion that comes back inconclusive means the traces did not arrive.

Plan Limits

LangWatch's free plan has limits on prompts, scenarios, evaluators, experiments, and datasets. When you hit a limit, the API returns "Free plan limit of N reached..." with an upgrade link.

How to handle:

• Work within the limits. If 3 resources of the relevant type are allowed, create 3 meaningful ones, not 10.
• Make every creation count: each one should demonstrate clear value.
• Show what works FIRST. If you hit a limit, summarize what was accomplished and note that upgrading the plan raises it. Point to the subscription settings on the platform, or to the license settings if LANGWATCH_ENDPOINT is set (self-hosted).
• Do NOT delete existing resources to make room or repurpose an existing resource to evade the limit.

Common Failures

Common Mistakes

• Do NOT weaken or remove the endpoint's existing authentication. The dedicated scenario key is an additional path, checked only when the request carries it.
• Do NOT put the raw key in the agent config. Store it with langwatch secret create and reference {{ secrets.SCENARIO_API_KEY }}.
• Do NOT skip trace adoption because the endpoint "already returns the answer". The reply text cannot prove a tool call happened; the trace can.
• Do NOT append a list of tools used to the response text so the judge can "see" them. That grades a self-report instead of evidence; adopt traceparent and the trace carries the real calls.
• Do NOT point the agent's tracing at a different LangWatch project than the one running the scenarios. The judge finds nothing there.
• Do NOT invent an agent id or pass a URL as the suite target. http: is followed by the Agent id from langwatch agent create or langwatch agent list --format json.
• Do NOT comma-separate --targets on suite create. It is space-separated and variadic; --scenarios is the comma-separated one.
• Do NOT guess the deployment URL or quietly default to localhost. Step 5 is a question for the user; ask and wait.
• Do NOT report success when a command failed. An unreachable platform or a failed run is part of the report, named per step.