Prometheus alerting rules that don't cry wolf

Write alerts on symptoms with for-durations and good labels so on-call gets pages that actually mean something.

Most alerting setups page too much and mean too little. The fix is to alert on symptoms users feel (error rate, latency, saturation), require them to persist with a for: duration, and label them so routing and silencing work. An alert that fires and resolves in 30 seconds is noise.

A rule that means something

groups:
  - name: api
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{code=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "5xx rate above 5% for 10m"

Symptoms, not causes

Route by severity

route:
  receiver: slack
  routes:
    - matchers: [severity="page"]
      receiver: pagerduty
    - matchers: [severity="ticket"]
      receiver: jira

Related posts