agent-observability-spec

1# Agent Observability Spec Skill
2 
3You can't fix what you didn't record. For LLM systems the unit of observability is the *trace* — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.
4 
5> **Not quite this?** Use `agent-spec` when you are specifying what the agent should do before it is built.
6 
7## What This Skill Produces
8 
9- A **trace schema**: per-request spans and the fields each must carry
10- **Metric definitions** across health, quality, cost, and behaviour — each with a threshold and owner
11- A **sampling and retention policy** that keeps cost sane and debugging possible
12- A **privacy note**: what logged content contains, who can see it, and how long it lives
13 
14## Required Inputs
15 
16Ask for (if not already provided):
17- **The system's shape** — single LLM call, RAG pipeline, or multi-step tool-using agent
18- **Traffic volume and cost sensitivity** — full tracing at 10M req/day is a budget decision
19- **What "misbehaving" means here** — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
20- **Existing observability stack** (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it
21 
22## Trace Schema
23 
24Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:
25 
26| Span | Must capture |
27|---|---|
28…