Prometheus alerting rules that don't cry wolf

Write alerts on symptoms with for-durations and good labels so on-call gets pages that actually mean something.

A Prometheus alert is a PromQL expression that fires when true for a configured duration. Most on-call pain comes not from missing alerts but from bad ones: paging on CPU > 80% for five seconds, labels with no service or severity, and annotations that say 'something is wrong' with no runbook link. Good alerting measures symptoms users feel — error rate, latency SLO burn, queue depth — not every internal gauge that twitching at 3am.

Name alerts after the user-visible failure mode (CheckoutPaymentErrorsHigh), not the cause (PodCpuHigh) unless cause-only pages are an explicit ops choice.

This note writes recording rules to simplify PromQL, alert rules with sensible for: windows, and routing labels Alertmanager needs. For the full observability stack — logs, metrics, traces — see Detection engineering; for log correlation when alerts fire, pair with Loki in the same course track.

Recording rules simplify alert PromQL

Expensive queries in every alert rule slow evaluation. Precompute job:http_requests:error_rate5m as a recording rule, then alert on the recorded metric. Keeps alert YAML readable and dashboards consistent with pages.

Multi-window burn-rate alerts (Google SRE style) combine short and long windows — page when both show budget burn, not on a single five-minute blip during deploy. Start simple with one for: duration; add burn rates when on-call trusts the baseline rules.

groups:
  - name: api_recording
    interval: 30s
    rules:
      - record: job:http_requests:error_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

Symptom-based alert with for duration

Use for: 5m (or longer) so brief spikes during deploys do not page. Set severity: page only for user-impacting conditions; severity: ticket for capacity warnings. Include $labels.job and $value in annotations so Slack messages are actionable without opening Grafana.

Alertmanager grouping (group_by: ['alertname', 'job']) batches related fires into one notification. Inhibition rules suppress symptom duplicates — disk full on node suppresses pod evicted on that node. Test routing with amtool alert add before changing production routes on a Friday.

groups:
  - name: api_alerts
    rules:
      - alert: HighErrorRate
        expr: job:http_requests:error_rate5m > 0.05
        for: 5m
        labels:
          severity: page
          team: platform
        annotations:
          summary: "High 5xx rate on {{ $labels.job }}"
          description: "Error rate {{ $value | humanizePercentage }} for 5m"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

Where this goes next

Wire Alertmanager routes to PagerDuty and Slack with inhibition rules — HighErrorRate suppresses PodRestarting on the same job. Correlate pages with Loki log queries in Grafana dashboards. Detection engineering covers metrics, logs, and alert design as one system.

Run alert retro meetings monthly: which pages led to action, which were false positives, which alerts had no runbook link. Delete or downgrade alerts that never drove a human response — they train on-call to ignore the pager.

Related posts