write-code-eval

1# Write a code evaluator
2 
3Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
4 
51. State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use `write-judge-prompt`.
62. Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
73. Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.
8 
9## Examples
10 
11| Failure mode | Possible check |
12|---|---|
13| Invalid output structure | Parse the output and check required fields |
14| Missing or forbidden text | Match a string or pattern |
15| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
16| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
17| Wrong value | Compare the output with a reference value |
18 
19Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.