eval-audit

1# Eval Audit
2 
3Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.
4 
5## Overview
6 
71. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
82. Run diagnostic checks across six areas
93. Produce a findings report ordered by impact, with each finding linking to a fix
10 
11## Prerequisites
12 
13Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."
14 
15## Connecting to Eval Infrastructure
16 
17Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.
18 
19## Diagnostic Checks
20 
21Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.
22 
23Prioritize findings by impact on the user's product. Present the most impactful findings first.
24 
25### 1. Error Analysis
26 
27**Check:** Has the user done systematic error analysis on real or synthetic traces?
28 
29Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.
30…