agent-evals

1# Agent Evals
2 
3Create repeatable checks so agent behavior improves safely over time.
4 
5## When to Use This Skill
6 
7Use this skill when:
8- Shipping new agent features or changing prompts
9- Adding CI gates for agent quality and safety
10- Building regression suites for tool-calling agents
11- Measuring LLM output quality at scale
12- Validating RAG retrieval accuracy
13 
14## Prerequisites
15 
16- Python 3.10+
17- An LLM API key (OpenAI, Anthropic, etc.)
18- pytest or a custom eval harness; `pytest-timeout` is required for the shown `--timeout` options
19- Optional: Braintrust, Promptfoo, or LangSmith account
20 
21## Evaluation contract
22 
23The following are application templates, not bundled executable tests. Supply the `agent` adapter, fixtures, supported provider/model IDs, reviewed dependencies and test configuration. Approve provider cost and data transmission; keep fixtures synthetic and keys private. Sandbox tool effects and inspect tool/network traces, not only response words. Keyword refusals and numerical judge scores do not prove safety or factual accuracy. Define thresholds and human review before evaluating.
24 
25## Evaluation Layers
26 
27### Unit Evals — Prompt-Level Correctness
28 
29Test individual prompt → response quality:
30 
31```python
32# evals/test_unit.py
33import json
34import pytest
35from agent import generate_response
36 
37CASES = json.load(open("evals/fixtures/unit_cases.json"))
38 
39@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
40def test_prompt_correctness(case):
41…