evaluate-rag

1# Evaluate RAG
2 
3## Overview
4 
51. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
62. Build a retrieval evaluation dataset: queries paired with relevant document chunks.
73. Measure retrieval quality with Recall@k (most important for first-pass retrieval).
84. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
95. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.
10 
11## Prerequisites
12 
13Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.
14 
15## Core Instructions
16 
17### Evaluate Retrieval and Generation Separately
18 
19Measure each component independently. Use the appropriate metric for each retrieval stage:
20 
21- **First-pass retrieval:** Optimize for Recall@k. Include all relevant documents, even at the cost of noise.
22- **Reranking:** Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.
23 
24### Building a Retrieval Evaluation Dataset
25 
26You need queries paired with ground-truth relevant document chunks.
27 
28**Manual curation (highest quality):** Write realistic questions and map each to the exact chunk(s) containing the answer.
29…