ai-agent-reliability

1# AI-Agent Reliability
2 
3An AI agent that works in a demo and one you can trust in production are different things — the gap is everything that happens when input is messy, the model hallucinates, a tool call goes wrong, or an error fails silently. This maps where your agent can fail and the specific checks that catch each, scaled to the stakes, plus a rollout that earns trust incrementally — so "works sometimes" becomes "works reliably."
4 
5## What This Skill Produces
6 
7- **A failure map** — where this agent can go wrong: bad/unexpected input, hallucinated output, wrong or malformed tool calls, unhandled edge cases, silent failures, and runaway loops
8- **The catching checks per failure** — input validation, output verification, evals on real cases, schema/format checks on tool calls, human-in-the-loop gates, and monitoring/alerts
9- **An eval approach** — testing on a real set of cases (including the hard ones) so quality is measured, not assumed, and regressions are caught
10- **Human-in-the-loop placement** — where a human must approve, scaled to consequence (irreversible/external actions gated, low-stakes automated)
11- **A right-sized plan** — reliability effort matched to the stakes, not gold-plating a low-risk toy or under-testing a high-risk system
12- **A trust-building rollout** — shadow mode → low-stakes → expand, with monitoring, rather than shipping it everywhere and hoping
13 
14## Required Inputs
15 
16Ask for these if not provided:
17…