write-judge-prompt

1# Write LLM-as-Judge Prompt
2 
3Design a binary Pass/Fail LLM-as-Judge evaluator for one specific failure mode. Each judge checks exactly one thing.
4 
5## Prerequisites
6 
7- Error analysis is complete. The failure mode is identified.
8- You have human-labeled traces for this failure mode (at least 20 Pass and 20 Fail examples).
9- A code-based evaluator cannot check this failure mode. Exhaust code-based options before reaching for a judge — many failure modes that seem subjective reduce to keyword checks, regex, or API calls when you understand the domain. Example: detecting whether an AI interviewing coach suggests "general" questions (asking about typical behavior instead of a specific past event) seems to require semantic understanding, but in practice a keyword check for words like "usually," "typical," and "normally" could work quite well.
10 
11## The Four Components
12 
13Every judge prompt requires exactly four components:
14 
15### 1. Task and Evaluation Criterion
16 
17State what the judge evaluates. One failure mode per judge.
18 
19```
20You are an evaluator assessing whether a real estate assistant's email
21uses the appropriate tone for the client's persona.
22```
23 
24Not: "Evaluate whether the email is good" or "Rate the email quality from 1-5."
25 
26### 2. Pass/Fail Definitions
27 
28Outcomes are strictly binary: Pass or Fail. No Likert scales, no letter grades, no partial credit. Define exactly what constitutes Pass and Fail. These definitions come from your error analysis failure mode descriptions.
29 
30```
31…