ai-sre-incident-response

1# AI SRE Incident Response
2 
3Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
4 
5## When to Use This Skill
6 
7- An LLM endpoint begins returning degraded or hallucinated answers
8- Token spend spikes beyond budget thresholds
9- A model provider goes down and traffic must fail over
10- Safety guardrails fire at abnormal rates
11- A new model deployment causes latency or accuracy regression
12 
13## Prerequisites
14 
15- Prometheus and Alertmanager deployed with scrape targets for AI services
16- Grafana dashboards for golden signals (latency, error rate, cost, quality)
17- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
18- Runbook repository accessible to responders
19- Rollback mechanism for model and prompt versions (GitOps or feature flags)
20 
21## AI Incident Classes
22 
23- **Availability incident**: model/provider unavailable, timeout storm.
24- **Quality incident**: answer accuracy or tool success drops below SLO.
25- **Safety incident**: harmful or policy-violating outputs increase.
26- **Cost incident**: unexpected token or provider spend spike.
27 
28## Severity Framework
29 
30| Severity | Criteria | Response Time | Notification |
31|----------|----------|---------------|--------------|
32| SEV1 | User-facing outage, compliance risk, data leak | 5 min | Page on-call + incident commander |
33| SEV2 | Major degradation in key flows | 15 min | Page on-call |
34| SEV3 | Limited impact or internal-only issue | 1 hour | Slack alert |
35…