Designing RocketRide Pipelines

Turns a plain-language request into an approved, lane-correct DAG — before any node is configured. Two outputs, two gates: a node selection (Gate A) and a wired topology (Gate B). The gate rules and forcing functions are in ../rocketride-building-pipelines/GATE_PROTOCOL.md — they apply here.

You work from the node index (L1): LAYER1_NODE_INDEX.json (bundled), or the live get_services() / .rocketride/services-catalog.json. Each entry is name · classType · lanes · invoke. The index is enough to select and wire; it carries no config fields (that's Phase 2). The bundled index is reconciled against this repo's node corpus (nodes/src/nodes/), so it can drift from the set a particular live engine actually has registered. The live MCP list_components tool lists names/summaries only (no lanes) and hides nodes whose integration isn't configured — use it to check availability, but wire from the bundled index + per-node schemas. If a node you expect is missing from it, check list_integrations first.

Phase 1a — Discover (→ Gate A)

1. Restate the task in one line: the input data, the wanted output. ("Input: user questions. Output: answers grounded in uploaded PDFs.")
2. Explore archetypes exhaustively. Group the index by classType and walk every archetype that could plausibly contribute. Don't stop at the first match. The archetypes: source (chat/webhook/dropper/filesys/telegram) · data/parse (parse/llamaparse/reducto/ landing_ai_parse/landing_ai_extract) · text (extract_data/ner/anonymize_text/dictionary/prompt/summarization) · preprocessor · image · audio · video · embedding · llm (14 providers) · store (vector DBs) · database (db_) · agent · tool (tool_) · memory · rerank · search · guard · infrastructure/target/response_* (terminals). For each relevant archetype, list the candidate nodes you see in the index.
3. Select, citing each: Found in index: <name> · classType=[…] · lanes={…}. A pipeline needs a resolvable source (one Source-mode node, or several with source naming the one that starts the run, in the file or at launch) and a terminal (response_* for a reply; a store / db_* for ingestion). Pull the right converters from the lane cheat-sheet in ../rocketride-building-pipelines/pipeline-patterns.md.
4. Gate A — end with the count line and the gate:

   Archetypes explored (N). Selected nodes (M): <name · role>, … — all cited from the index. Approve these M nodes? (yes / adjust / cancel) Then use the Write tool to write .context/GATE_STATE.md = GATE A | presented | status: AWAITING (GATE_PROTOCOL §2 — so the gate survives a context reset), and STOP. (See §1.)

Phase 1b — Design the DAG (→ Gate B)

Only after Gate A is approved.

1. Fetch each selected node's schema (L2 — describe_component / tools/ shim in ../rocketride-configuring-pipelines/tools/fetch-node-schema.py / .rocketride/schema/<n>.json). You need the real lane signatures here, not the index summary — wiring from the summary is the most common silent bug. (Forcing function 8.) Lazily — only for the nodes you selected at Gate A; never ls/cat/glob the whole .rocketride/schema/ dir or "pull every schema up front" (FF#17). Select from the index first; fetch only the few schemas you need.
2. Wire the lanes. Each non-source node gets input: [{lane, from}]. State every edge with its lane type: chat_1 → embedding_1 (lane: questions). The output lane of from must be a real output of that node and a valid input of the target. If types don't match, insert a converter (cheat-sheet) — don't force it.
   • <Bad>: parse_1 → store_1 (lane: text), wiring parsed text straight into a vector store. It passes the eye test but fails validation: a store ingests documents that already carry embedding vectors, and text is not one of its lanes.
   • <Good>: parse_1 → preprocessor_langchain_1 (lane: text), then preprocessor_langchain_1 → embedding_1 (lane: documents), then embedding_1 → store_1 (lane: documents).
3. Apply the structural rules (PIPELINE_RULES_SUMMARY.md): a resolvable source (rule 1); acyclic (no loops); no orphans (every node reachable from the source); embedding before any store; agents wire their llm/tool/memory via the control plane (the controlled node carries control: [{classType, from: <agent>}], the agent does not) and must meet each invoke min/max from the index.
4. Draw it (Mermaid or ASCII) and Gate B:

   . Edges (K): <node → node (lane: type)>, … — all lane-checked against schemas. Approve this topology? (yes / adjust / cancel) Then Write .context/GATE_STATE.md = GATE B | presented | status: AWAITING (§2), and STOP.

Hand the approved selection + topology to rocketride-configuring-pipelines.

Red flags

Supporting files

• LAYER1_NODE_INDEX.json — the thin node index (name · classType · lanes · invoke)
• PIPELINE_RULES_SUMMARY.md — lane types, lane-transform table, structural + control-plane rules
• examples/ — worked pipelines (simple-chat-rag, document-ingestion, agentic-chat) in examples/README.md + a FAILURE_SCENARIOS.md of what not to do
• deep docs — when a node is unfamiliar or you're unsure how an archetype behaves, fetch ONE page: ../rocketride-building-pipelines/tools/fetch-doc.py "<node-name>" (→ /nodes/<name>.md) or … "execution model" (→ /concepts/execution-model.md for lanes). Never llms-full.txt.