bootstrap

1<HARD-GATE>
2NO v0 grader deployed WITHOUT explicitly marking it as uncalibrated.
3NO synthetic labels — LLM can generate eval inputs, but labels MUST come from real system output + human judgment.
4NO principle without a source label documenting where it came from.
5</HARD-GATE>
6 
7# Bootstrap
8 
9Cold-start an evaluation system when you have nothing. In 30 minutes you get a working
10v0 grader and a clear path to a calibrated, trustworthy evaluation.
11 
12> **Requires OpenJudge** (`pip install py-openjudge`) for the grader generators
13> (`SimpleRubricsGenerator` / `IterativeRubricsGenerator`). The interview, stratification,
14> and calibration-roadmap methodology is SDK-independent.
15 
16## Checklist
17 
18You MUST create a task for each item and complete them in order:
19 
201. **Understand the product** — one-shot interview, not question-by-question
212. **Generate v0 grader** — use OpenJudge SimpleRubricsGenerator
223. **Synthesize eval inputs** — 30 inputs with 60/30/10 stratification
234. **Run v0 evaluation** — GradingRunner with the generated grader
245. **Output roadmap** — exactly how to reach 50 labels → calibrate
25 
26## Step 1: Product Interview (One Shot)
27 
28Ask the user to describe their system in one go:
29 
30```
31To bootstrap your evaluation, I need to understand what you're building.
32Please describe (all at once):
33 
34- What does your system do? Who uses it?
35- What are 3 examples of perfect outputs?
36- What are 3 things the system must never do?
37- What failures worry you most?
38```
39…