eval-design

1# Eval Design
2 
3Design high-quality evaluation datasets that measure what actually matters for your
4application. You extract evaluation dimensions from business context, structure them
5into stratified test cases, and output datasets ready for OpenJudge `GradingRunner`.
6 
7## When to Activate
8 
9- User has agent traces / production logs and wants to build an eval set from them
10- User has evaluation principles but needs properly stratified test data
11- User wants to generate adversarial examples that stress-test their system
12- User needs coverage analysis — are they testing all the right things?
13- User wants a labeling guide for human annotators
14 
15## Checklist
16 
17You MUST create a task for each item and complete them in order:
18 
191. **Extract eval dimensions** — from traces, spec, or user interview
202. **Design stratified sampling** — 60/30/10 split with difficulty strata
213. **Generate test data** — synthetic inputs + adversarial examples
224. **Output OpenJudge dataset** — structured format ready for GradingRunner
23 
24## Coverage check: run the bundled script
25 
26After you have a dataset, validate coverage with the bundled, tested script
27(`scripts/coverage_check.py`, standard library only, **no OpenJudge dependency**) before
28trusting any per-slice metric:
29 
30```bash
31python scripts/coverage_check.py --dataset eval-data/dataset.jsonl
32```
33 
34It reports per-dimension and per-(dimension × stratum) counts, flags thin cells
35…