agent-evals

1# Agent Evals
2 
3Create repeatable checks so agent behavior improves safely over time.
4 
5## When to Use This Skill
6 
7Use this skill when:
8- Shipping new agent features or changing prompts
9- Adding CI gates for agent quality and safety
10- Building regression suites for tool-calling agents
11- Measuring LLM output quality at scale
12- Validating RAG retrieval accuracy
13 
14## Prerequisites
15 
16- Python 3.10+
17- An LLM API key (OpenAI, Anthropic, etc.)
18- pytest or a custom eval harness
19- Optional: Braintrust, Promptfoo, or LangSmith account
20 
21## Evaluation Layers
22 
23### Unit Evals — Prompt-Level Correctness
24 
25Test individual prompt → response quality:
26 
27```python
28# evals/test_unit.py
29import json
30import pytest
31from agent import generate_response
32 
33CASES = json.load(open("evals/fixtures/unit_cases.json"))
34 
35@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
36def test_prompt_correctness(case):
37    result = generate_response(case["prompt"], model=case.get("model", "default"))
38    # Exact match for structured output
39    if case.get("expected_json"):
40        assert json.loads(result) == case["expected_json"]
41    # Substring match for free-text
42    for keyword in case.get("must_contain", []):
43        assert keyword.lower() in result.lower(), f"Missing: {keyword}"
44    for keyword in case.get("must_not_contain", []):
45        assert keyword.lower() not in result.lower(), f"Unexpected: {keyword}"
46```
47 
48Golden dataset format:
49 
50```json
51[
52  {
53    "id": "calc-01",
54…