alerting-oncall

1# Alerting & On-Call
2 
3Configure effective alerting and on-call management for production systems.
4 
5## When to Use This Skill
6 
7Use this skill when:
8- Setting up alerting rules and thresholds
9- Configuring on-call rotations and schedules
10- Implementing alert routing and escalation
11- Reducing alert fatigue
12- Managing incident response workflows
13 
14## Prerequisites
15 
16- Monitoring system (Prometheus, Datadog, etc.)
17- On-call platform (PagerDuty, Opsgenie, Grafana OnCall)
18- Communication channels (Slack, email)
19 
20## Alerting Best Practices
21 
22### Alert Categories
23 
24```yaml
25# Severity levels
26critical:
27  - Service completely down
28  - Data loss imminent
29  - Security breach
30  response: Immediate page, wake people up
31 
32high:
33  - Service degraded significantly
34  - Error rate above SLO
35  - Capacity near limit
36  response: Page during business hours, notify after hours
37 
38medium:
39  - Performance degradation
40  - Non-critical component failure
41  - Warning thresholds exceeded
42  response: Notify via Slack, review next business day
43 
44low:
45  - Informational alerts
46  - Capacity planning triggers
47  - Routine maintenance needed
48  response: Email notification, weekly review
49```
50 
51### Alert Design Principles
52 
53```yaml
54# Good alert characteristics
55alerts:
56  actionable:
57    - Every alert should require human action
58    - Include runbook links
59    - Clear remediation steps
60 
61  relevant:
62    - Alert on symptoms, not causes
63    - Focus on user impact
64    - Avoid alerting on expected behavior
65 
66  timely:
67…