prompt-regression

1<HARD-GATE>
2NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported.
3NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison.
4NO comparison with fewer than 10 samples per axis — CI is too wide to be meaningful.
5</HARD-GATE>
6 
7# Prompt Regression
8 
9Compare two prompts head-to-head and determine, with statistical rigor, whether
10the candidate is better, worse, or tied on each evaluation dimension.
11 
12## When to Activate
13 
14- You changed the system prompt and want to verify it's actually better
15- You're iterating on RAG answer templates
16- You're optimizing agent step-by-step instructions
17- You want data to support a prompt change decision
18 
19## Checklist
20 
21You MUST create a task for each item and complete them in order:
22 
231. **Load and analyze prompts** — diff the baseline vs candidate
242. **Derive comparison dimensions** — from the prompt changes + task type
253. **Select graders per dimension** — pairwise, judge, or rule
264. **Run position-debiased comparison** — swap-aggregate to eliminate order bias
275. **Compute statistics** — win rates + bootstrap 95% CI per dimension
286. **Present results** — per-dimension verdict with confidence intervals
29 
30## Fast path: run the bundled script
31 
32Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
33wrong). Run the bundled, tested script (`scripts/pairwise.py`, standard library only, **no
34OpenJudge dependency**):
35 
36```bash
37…