evals-bootstrap

1# Bootstrap the eval suite
2 
3Theory: [Two kinds of checks](https://secondbrainos.dev/#/read/course-5-evals/two-kinds-of-checks)
4and the full walkthrough in [Evals practice](https://secondbrainos.dev/#/read/course-5-evals/evals-practice).
5An eval suite is the same test after every change. Behavioural checks read
6the steps of a trace; end-to-end checks read only the result. Start
7behavioural: they are deterministic, run in seconds, and diagnose instead of
8just scoring.
9 
10## Workflow
11 
121. **Find real failures.** Ask where the agent's runs live (logs, transcripts,
13   a traces directory). Read until you have up to twenty real failures — not
14   imagined ones. If there are no logged runs yet, build the trace logging
15   first (step 3) and seed the suite with the three failures the user can
16   recall; a small honest suite beats a large invented one.
172. **One line per failure.** For each: the input, and the one specific
18   behaviour that should have happened and did not. Failures cluster into
19   four to eight behaviours; name them.
203. **Ensure traces exist.** Each run must be stored as `traces/<id>.json` —
21   a list of events including tool calls. If the user's harness is Claude
22   Code, the transcript already is the trace; wire up whatever copies or
23   converts it. No trace, no behavioural checks.
244. **Write `cases.yaml`.** One entry per failure:
25 
26```yaml
27- id: refund_1042
28  input: "Refund order #1042, customer says it arrived broken"
29…