eval-engineering

1# Eval Engineering
2 
3## Flow
4 
5- Inspect all inputs first: the repository, Harness, optional traces, existing
6  Tasks and runs, existing World knowledge, and the human goal. Identify the
7  source files and skill references that apply before proposing work.
8- Create or update the small project World Knowledge Skill from reusable facts
9  in those inputs. Use it to propose one grounded Task.
10- Draft the Task Spec and World Skill together. Show both to the user, keep exact
11  Task truth only in `Task.md`, and refine both until the user approves them.
12- Implement the approved Task, validate its Environment and Verifier, run the
13  real Harness, inspect the full evidence, and fix only non-agent failures.
14- Reconcile the World Skill with what the run proved, then repeat this flow for
15  the next Task.
16 
17## Terms
18 
19- **Task Spec:** `Task.md`, which describes the input, relevant agent
20  conditions, Environment, scoring, fairness, and open decisions for one Task.
21- **Task:** the runnable instruction, Environment, and Verifier.
22- **Harness:** the complete agent Harbor runs, including prompts, model loop,
23  tools, hooks, memory, sessions, and adapter.
24- **Environment:** the files, data, services, identity, permissions, network,
25  clock, and mutable state around the Harness.
26- **Verifier:** independent checks that score the result or mark a run invalid.
27- **World Knowledge Skill:** a repository-local skill with reusable
28…