Observability & SRE interview questions

Metrics (aggregated numeric time series), logs (discrete timestamped events), and traces (the path of a request across services). Together they let you both detect a problem and explain it.

Monitoring watches known failure modes with predefined dashboards and alerts; observability is the property that lets you ask new questions about unknown failures from your telemetry without shipping new code. One is a subset of good practice, the other a system property.

Monitoring answers questions you knew to ask (is CPU high?); observability lets you answer ones you did not (why are only Android users in region X slow?). You get there with high-cardinality, well-structured telemetry — not more dashboards.

Cardinality is the number of unique label combinations on a metric. Each combination is a separate time series, so a high-cardinality label (user ID, request ID) can explode series count and blow up memory and query cost — the number-one operational risk with metrics.

RED (Rate, Errors, Duration) describes request-driven services from the user’s view; USE (Utilization, Saturation, Errors) describes resources like CPU, disk, and queues. Use RED for services, USE for infrastructure.

They answer different questions: RED tells you the user-visible health of a service; USE tells you whether a resource is the bottleneck. On an incident you usually start at RED (are users hurting?) and drill to USE (what resource is saturated?).

Latency, traffic, errors, and saturation — Google SRE’s minimal set for user-facing systems. If you can only watch four things, these catch most user-impacting problems.

It scrapes HTTP /metrics endpoints on a schedule (pull model), discovering targets via service discovery (Kubernetes, Consul). Push is only for short-lived jobs via the Pushgateway.

Pull means Prometheus owns the schedule and instantly knows when a target is down (a failed scrape), which push cannot tell you. Targets just expose /metrics; service discovery keeps the target list current as pods come and go.

Counter (monotonic, e.g. requests_total), Gauge (up/down, e.g. memory), Histogram (bucketed observations for latency, enabling quantiles), and Summary (client-side quantiles). Choosing the right type drives what queries are possible.

http_requests_total                       # counter (only rises)
node_memory_MemAvailable_bytes            # gauge (up/down)
http_request_duration_seconds_bucket      # histogram (buckets)

Take the ratio of 5xx request rate to total request rate over a window. rate() turns a counter into per-second change; sum() aggregates across instances/labels.

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

With a histogram, use histogram_quantile over rate of the _bucket series (e.g. p99 latency). Percentiles must come from histograms — you cannot average pre-aggregated averages to get an accurate tail.

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Recording rules precompute expensive queries into new series for fast dashboards and reuse; alerting rules evaluate a condition over time and fire to Alertmanager, which handles grouping, routing, silencing, and deduplication.

- alert: HighErrorRate
  expr: job:error_rate:ratio5m > 0.02
  for: 10m
  labels: { severity: page }
  annotations: { summary: "5xx rate over 2%" }

It groups related alerts, deduplicates identical ones from many instances, routes by label to the right receiver (page vs ticket), and supports silences and inhibition so a known outage does not spam every downstream alert.

Emitting logs as key/value JSON makes them queryable, filterable, and aggregatable, instead of grepping free text. It also enables correlation via consistent fields like request_id, service, and level.

{"ts":"2026-07-04T10:20:01Z","level":"error",
 "service":"api","trace_id":"a1b2c3",
 "msg":"db timeout","dur_ms":812}

Following one request across services as a trace made of spans, each with timing and parent/child links propagated via context headers. It shows where latency and errors occur across a call graph that logs alone cannot reveal.

Each service creates a span and passes a trace context header (traceparent) to the next, so the backend can stitch spans into one waterfall. That is how you see that a slow request spent 400ms in the payments call, not the gateway.

A vendor-neutral standard and SDK/collector for generating and exporting metrics, logs, and traces. It decouples instrumentation from the backend, so you can switch vendors without re-instrumenting code.

You instrument once against the OTel API and run a Collector that receives, processes (batching, sampling, redaction), and exports to any backend. Swapping Jaeger for a SaaS vendor is a Collector config change, not a code change.

Thread a shared trace_id through all three: put it on structured logs, attach it to metric exemplars, and it is native to spans. Then a spike on a latency graph links to the exact slow trace and its logs.

Tracing every request is expensive at scale, so you sample — head-based (decide up front) or tail-based (keep slow/error traces after seeing the whole trace). Tail sampling keeps the interesting traces while controlling cost.

Head sampling is cheap but blind — it may drop the one errored request you needed. Tail sampling buffers spans and decides after the trace completes, so you keep all errors and slow outliers and drop the boring fast successes. The cost is memory in the Collector.

An SLI is a measured indicator (e.g. success rate); an SLO is the target for it (99.9%); the error budget is 1 minus the SLO — the allowed unreliability you can spend on shipping features before you must slow down and stabilize.

The error budget turns reliability into a shared, quantified decision: while budget remains you ship features; when you burn it, the policy is to stop and stabilize. It aligns dev and ops on one number instead of arguing about whether things are stable enough.

SLO: 99.9% of requests succeed over 30d
Error budget: 0.1% = ~43m of downtime / 30d
Burn 43m of failures -> feature freeze until recovered

Instead of alerting on a raw threshold, you alert on how fast the error budget is being consumed over multiple windows (e.g. fast burn over 5m and slow burn over 1h). It pages only when the budget is genuinely at risk, cutting noise.

Alert on symptoms that affect users (SLO burn rate) rather than every cause, page only for actionable and urgent conditions, route the rest to dashboards/tickets, and tune or delete noisy alerts ruthlessly.

Every page should be urgent, actionable, and tied to user impact; if it is none of those it should be a ticket or a dashboard, not a 3am page. Track pages per on-call shift and delete or retune the noisiest — alert hygiene is ongoing work, not a one-off.

Detect and declare, assign a clear incident commander, mitigate first (roll back, shed load) before root-causing, communicate status regularly, then resolve and write a blameless postmortem. Restoring service beats understanding it.

Blameless, focused on systemic causes and contributing factors, with a clear timeline, customer impact, and concrete action items with owners. The goal is learning and prevention, not attributing fault to a person.

Blameless because humans act reasonably given the system they are in — the fix is the system, not the person. A good doc has a factual timeline, quantified impact, the contributing factors, and action items with owners and due dates that actually get tracked.

Avoid unbounded labels (user IDs, request IDs) on metrics, aggregate at the right level, use recording rules and retention/downsampling, and push per-request detail to logs/traces instead. Cardinality is the main driver of TSDB cost and slowness.

# BAD: user_id is unbounded -> millions of series
http_requests_total{user_id="..."}
# GOOD: bounded labels; put user_id in logs/traces
http_requests_total{route="/api/x", status="200"}