Generate a RAG Evaluation Dataset

This recipe analyzes your RAG knowledge base and generates a comprehensive Q&A evaluation dataset.

Step 1: Analyze the Knowledge Base

Read the codebase to find the knowledge base:

• Document files (PDFs, markdown, text files)
• Database schemas (if documents are stored in a DB)
• Vector store configuration (what's being embedded)
• Chunking strategy (how documents are split)

Read every document you can access. Understand:

• What topics does the knowledge base cover?
• What's the depth of information?
• What terminology is used?
• What are the boundaries (what's NOT covered)?

Step 2: Generate Diverse Question Types

Create questions across these categories:

Factual Recall

Direct questions answerable from a single passage:

• "What is the recommended threshold for X?"
• "When should Y be applied?"

Multi-Hop Reasoning

Questions requiring information from multiple passages:

• "Given condition A and condition B, what should be done?"
• "How do X and Y interact when Z occurs?"

Comparison

Questions comparing concepts within the knowledge base:

• "What's the difference between approach A and approach B?"
• "When should you use X instead of Y?"

Edge Cases

Questions about boundary conditions or unusual scenarios:

• "What happens if the measurement is outside normal range?"
• "What if two recommendations conflict?"

Negative Cases

Questions about topics NOT covered by the knowledge base:

• "Does the system support Z?" (when it doesn't)
• Questions requiring external knowledge the KB doesn't have

These help test that the agent correctly says "I don't know" rather than hallucinating.

Step 3: Include Context Per Row

For each Q&A pair, include the relevant document chunk(s) that contain the answer. This enables:

• Platform experiments without the full RAG pipeline
• Evaluating answer quality independent of retrieval quality
• Testing with different prompts using the same retrieved context

Format:

{
    "input": "When should I irrigate apple orchards?",
    "expected_output": "Irrigate to maintain soil moisture between 25-35 kPa...",
    "context": "## Irrigation Management\nSoil moisture threshold for apple orchards: maintain between 25-35 kPa...",
    "question_type": "factual_recall"
}

Step 4: Export Formats

Create both:

Python DataFrame (for SDK experiments)

import pandas as pd
df = pd.DataFrame(dataset)
df.to_csv("rag_evaluation_dataset.csv", index=False)

Platform-Ready CSV

Export with columns: input, expected_output, context, question_type This can be imported directly into LangWatch platform datasets.

Step 5: Validate Dataset Quality

Before using the dataset:

1. Check topic coverage: are all knowledge base topics represented?
2. Verify answers are actually in the context, with no hallucinated expected outputs
3. Check question diversity: not all the same type
4. Verify negative cases have appropriate "I don't know" expected outputs
5. Run a quick experiment to baseline accuracy

Common Mistakes

• Do NOT generate questions without reading the actual knowledge base first
• Do NOT skip negative cases. Testing "I don't know" is crucial for RAG
• Do NOT use the same question pattern for every entry. Diversify types
• Do NOT forget to include the relevant context per row
• Do NOT generate expected outputs that aren't actually in the knowledge base