rag-eval

1# RAG Eval
2 
3Evaluate RAG systems by diagnosing retrieval and generation separately. A single
4"RAG accuracy" number hides whether the problem is finding the right documents or
5using them correctly. This skill separates them so you know what to fix.
6 
7## When to Activate
8 
9- User has a RAG pipeline (retriever + generator) with traces
10- User wants to know if their RAG system hallucinates
11- User is optimizing chunking strategy and needs before/after comparison
12- User wants to build a RAG evaluation dataset
13 
14## Checklist
15 
16You MUST create a task for each item and complete them in order:
17 
181. **Load RAG traces** — validate query + context + answer triples
192. **Separate retrieval vs generation** — determine which layers to evaluate
203. **Run faithfulness evaluation** — is the answer grounded in retrieved docs?
214. **Run retrieval evaluation** — are the right documents retrieved?
225. **Build diagnostic matrix** — cross-tabulate to find root cause
236. **Output findings** — prioritized issues with concrete fixes
24 
25## Fast path: run the bundled script
26 
27Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the
28retrieval-vs-generation diagnostic matrix with the bundled, tested script
29(`scripts/rag_diagnostic.py`, standard library only, **no OpenJudge dependency**):
30 
31```bash
32python scripts/rag_diagnostic.py --traces traces.jsonl
33```
34 
35Trace rows: `{"faithful":bool}` or `{"faithfulness_score":1-5}` (>= 4 = faithful), plus an
36…