metric-design

1# Metric Design
2 
3Select, configure, and combine evaluation graders into a working pipeline. You choose
4the right tool for each evaluation dimension — from zero-cost code checks to LLM judges
5— and produce executable `GradingRunner` code that runs on OpenJudge.
6 
7> **Requires OpenJudge** (`pip install py-openjudge`). This skill is intentionally
8> SDK-centric — grader selection, `GradingRunner`, and aggregators are OpenJudge APIs. The
9> design/decision logic still applies if you use another harness; only the code does not.
10 
11## When to Activate
12 
13- User has eval dimensions/principles but doesn't know which grader type to use
14- User wants to write an LLM-as-judge prompt for a specific failure mode
15- User needs a composite score combining multiple evaluation dimensions
16- User wants to auto-generate graders from labeled data instead of writing them manually
17- User's current evaluation is all LLM-based and too expensive/too slow
18 
19## Checklist
20 
21You MUST create a task for each item and complete them in order:
22 
231. **Select grader types** — per dimension, pick the right grader class
242. **Create custom graders** — write judge prompts (4-component) or function graders
253. **Auto-generate if applicable** — use OpenJudge Generator for cold starts
264. **Run anti-pattern scan** — check for Likert, missing few-shot, vague criteria
275. **Build pipeline code** — assemble GradingRunner with graders + aggregators
28 
29## Step 1: Select Grader Type Per Dimension
30…