agentic-eval

1# Agentic Evaluation Patterns
2 
3Patterns for self-improvement through iterative evaluation and refinement.
4 
5## Overview
6 
7Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.
8 
9```
10Generate → Evaluate → Critique → Refine → Output
11    ↑                              │
12    └──────────────────────────────┘
13```
14 
15## When to Use
16 
17- **Quality-critical generation**: Code, reports, analysis requiring high accuracy
18- **Tasks with clear evaluation criteria**: Defined success metrics exist
19- **Content requiring specific standards**: Style guides, compliance, formatting
20 
21---
22 
23## Pattern 1: Basic Reflection
24 
25Agent evaluates and improves its own output through self-critique.
26 
27```python
28def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
29    """Generate with reflection loop."""
30    output = llm(f"Complete this task:\n{task}")
31    
32    for i in range(max_iterations):
33        # Self-critique
34        critique = llm(f"""
35        Evaluate this output against criteria: {criteria}
36        Output: {output}
37        Rate each: PASS/FAIL with feedback as JSON.
38        """)
39        
40        critique_data = json.loads(critique)
41        all_pass = all(c["status"] == "PASS" for c in critique_data.values())
42        if all_pass:
43            return output
44        
45        # Refine based on critique
46…