align-human

1<HARD-GATE>
2NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%.
3NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6.
4NO alignment conclusion WITHOUT all 5 bias checks completed.
5</HARD-GATE>
6 
7# Align Human
8 
9Measure whether your automatic judge agrees with human judgment, detect where and
10why they disagree, and build a roadmap to reduce human review over time.
11 
12## When to Activate
13 
14- You have a working judge/grader and 50+ human-labeled examples
15- You want to know if the judge is trustworthy enough to replace human review
16- You've noticed the judge's decisions being overturned by humans
17- You're preparing to deploy an evaluation as a production gate
18 
19## Checklist
20 
21You MUST create a task for each item and complete them in order:
22 
231. **Load paired data** — match judge verdicts with human labels
242. **Measure TPR/TNR** — confusion matrix + per-stratum breakdown
253. **Calculate agreement** — Cohen's kappa, Gwet's AC1, systematic bias
264. **Run bias detection** — 5 systematic bias checks
275. **Analyze disagreements** — cluster patterns + diagnose root causes
286. **Build human-reduction roadmap** — 4-phase transition plan
297. **Confirm and record** — one confirmation, then write results
30 
31## Fast path: run the bundled script
32 
33Don't hand-write the calibration statistics — that is exactly where subtle bugs hide. Run
34…