agent-observability

1# Agent Observability
2 
3Monitor AI agent behavior with logs, traces, metrics, and cost telemetry. This skill covers the full observability stack for LLM-powered applications: from raw Prometheus counters to Grafana dashboards, OpenTelemetry tracing, structured logging, cost tracking, SLO definition, and PII redaction.
4 
5---
6 
7## When to Use
8 
9Apply this skill whenever you operate:
10 
11- **Autonomous AI agents** that make multi-step tool calls (e.g., coding agents, support agents, data-pipeline agents).
12- **LLM-backed APIs** serving chat completions, summarisation, or classification behind a REST or gRPC gateway.
13- **RAG pipelines** where a retriever fetches context from a vector store before prompting a model.
14- **Multi-agent orchestrations** (crew-style or graph-based) where several agents collaborate on a single task.
15- **Batch inference jobs** that process thousands of prompts against a model endpoint.
16 
17Key signals that you need this skill:
18 
191. You cannot answer "what is p95 latency for agent responses this week?"
202. You have no per-request cost attribution.
213. Debugging a bad agent response requires grepping raw application logs.
224. You have no alerting on token-usage spikes or elevated error rates.
23 
24---
25 
26## Core Metrics
27 
28Define these metrics at the application layer. All examples use the Prometheus client library naming conventions.
29 
30### Latency
31 
32```python
33from prometheus_client import Histogram
34 
35# Total end-to-end latency for a full agent turn (user prompt -> final response)
36…