validate-evaluator

1# Validate Evaluator
2 
3Calibrate an LLM judge against human judgment.
4 
5## Overview
6 
71. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
82. Run judge on dev set and measure TPR/TNR
93. Iterate on the judge until TPR and TNR > 90% on dev set
104. Run once on held-out test set for final TPR/TNR
115. Apply bias correction formula to production data
12 
13## Prerequisites
14 
15- A built LLM judge prompt (from write-judge-prompt)
16- Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
17  - Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
18  - Labels must come from a domain expert, not outsourced annotators
19- Candidate few-shot examples from your labeled data
20 
21## Core Instructions
22 
23### Step 1: Create Data Splits
24 
25Split human-labeled data into three disjoint sets:
26 
27| Split | Size | Purpose | Rules |
28|-------|------|---------|-------|
29| **Training** | 10-20% (~10-20 examples) | Source of few-shot examples for the judge prompt | Only clear-cut Pass and Fail cases. Used directly in the prompt. |
30| **Dev** | 40-45% (~40-45 examples) | Iterative evaluator refinement | Never include in the prompt. Evaluate against repeatedly. |
31| **Test** | 40-45% (~40-45 examples) | Final unbiased accuracy measurement | Do NOT look at during development. Used once at the end. |
32…