rag-observability-evals

1# RAG Observability and Evaluations
2 
3Run retrieval-augmented generation like a measurable production system, not a black box.
4 
5## When to Use This Skill
6 
7- Deploying a RAG system to production and need quality monitoring
8- Setting up automated evaluation pipelines for retrieval and generation
9- Debugging hallucination or relevance regressions
10- Building dashboards for RAG-specific golden signals
11- Establishing quality gates for RAG pipeline changes
12 
13## Prerequisites
14 
15- RAG pipeline with instrumented retrieval and generation stages
16- Python 3.10+ with evaluation libraries (ragas, langchain, openai)
17- Prometheus endpoint for custom metrics export
18- Benchmark dataset with gold-standard question/answer/source triples
19- OpenTelemetry SDK integrated into the RAG service
20 
21## What to Measure
22 
23### Retrieval Quality
24- Recall@k and MRR for top-k chunks
25- Citation coverage and source freshness
26- Embedding drift and index staleness
27 
28### Generation Quality
29- Groundedness score (answer supported by retrieved context)
30- Hallucination rate by route/use case
31- Instruction adherence and format validity
32 
33### Reliability and Cost
34- p50/p95 latency split by retrieval vs generation
35- Token usage per stage
36- Cache hit rate and cost per successful answer
37 
38## RAGAS Evaluation Script
39 
40```python
41# rag_eval.py
42"""Evaluate RAG pipeline quality using RAGAS metrics."""
43from ragas import evaluate
44from ragas.metrics import (
45    faithfulness,
46    answer_relevancy,
47    context_precision,
48    context_recall,
49…