evaluations
GitHubLangWatch评估请求的路由技能，用于区分预部署实验与生产在线评估。通过意图分类将请求转发至对应子技能，若歧义则引导用户选择。
Trigger Scenarios
Install
npx skills add langwatch/langwatch --skill evaluations -g -y
SKILL.md
Frontmatter
{
"name": "evaluations",
"license": "MIT",
"description": "Compatibility router for LangWatch evaluation requests. Use only when the user asks for evaluations without making it clear whether they mean pre-deployment experiments or production online evaluations. Routes the request to the focused companion skill and does not implement either workflow itself.",
"user-prompt": "Help me evaluate my agent",
"compatibility": "Works with Claude Code and similar AI assistants. The `langwatch` CLI is the only interface for platform operations and documentation."
}
Route an Evaluation Request
This is a compatibility skill. Do not build an experiment, monitor, or guardrail from this skill.
Classify the user's intent:
| Intent | Correct skill |
|---|---|
| Batch test a dataset, compare prompts or models, benchmark, create a CI quality gate | experiments |
| Score live traces or threads, monitor production quality, create a guardrail | online-evaluations |
If the request remains ambiguous after inspecting context (a bare "make me an eval" that names neither a dataset nor live traffic), do not create anything yet. This choice picks what gets tested, so it is the user's to make, not a default's. Ask it as a question card and stop; the answer arrives as the next message.
Where langy-card blocks render, ask it as a choices block (the only sanctioned question format) last in the reply:
```langy-card
{
"kind": "choices",
"blockId": "eval-kind",
"question": "What should this evaluate?",
"options": [
{ "id": "experiment", "label": "A dataset, before deployment" },
{ "id": "online", "label": "Live production traffic" }
]
}
```
Neither option names an existing entity, so neither carries a ref. Without that channel, ask the same question as one short line of prose.
A rejected field value is not this kind of choice. If a create later fails with a validation_error whose reason names the field and an expected list, correct that exact field from the list and retry once. Never turn a fixable slug into a question for the user.
Then hand off:
-
If the correct companion skill is available, load it and follow it instead of continuing here.
-
If
experimentsis missing, tell the user to install it with:npx skills@1.5.19 add langwatch/skills/experiments -
If
online-evaluationsis missing, tell the user to install it with:npx skills@1.5.19 add langwatch/skills/online-evaluations
Do not recreate the companion skill's instructions from memory. Load the focused skill so its current workflow, safety checks, and verification steps are used.
Version History
- 12615f1 Current 2026-08-20 10:01
