Observability & SRE interview questions

The short version is metrics, logs, and traces. Metrics are numeric time series, logs are discrete events, traces follow a request across services. Together they tell you that something's wrong, where, and often why.

metrics → detect (error rate spike)
traces  → locate (slow dependency span)
logs    → explain (timeout message + fields)

The short version is monitoring watches known failure modes with predefined alerts and dashboards. Observability is the property that lets you ask new questions about unknown failures from the telemetry you already have — without shipping new code.

monitoring:    is CPU > 90%?
observability: why are only Android users in region X slow?

Latency, traffic, errors, and saturation — Google SRE's minimal set for user-facing systems. If I could only watch four things, these catch most of the user-impacting problems.

latency   → p50/p99 request duration
traffic   → requests per second
errors    → 5xx / success ratio
saturation→ queue depth, CPU, thread pool

RED — Rate, Errors, Duration — is the user's view of a request-driven service. USE — Utilization, Saturation, Errors — is for resources like CPU, disk, and queues. I start with RED for user impact, then USE to find the bottleneck.

On an incident I usually ask 'are users hurting?' with RED before 'which resource is maxed?' with USE. Mixing them on one graph confuses responders. Microservices get RED dashboards per service; node and DB tiers get USE. Saturation — queue depth, thread pool wait — often predicts latency before utilization hits 100%.

api service:  rate / error ratio / latency histogram
postgres host: CPU util / runnable queue / disk errors

Cardinality is how many unique label combinations a metric has. Each combo is a time series — so unbounded labels like user_id or request_id explode memory, query cost, and scrape time.

# BAD — unbounded
http_requests_total{user_id="..."}
# GOOD — bounded
http_requests_total{route="/api/pay", code="500"}
# put user_id on logs/traces instead

I'd rather emit logs as key/value JSON so I can filter and aggregate instead of grepping free text. Consistent fields — service, level, trace_id — are what let you correlate with metrics and traces.

{"ts":"2026-07-24T10:20:01Z","level":"error",
 "service":"api","trace_id":"a1b2c3",
 "msg":"db timeout","dur_ms":812}

Prometheus scrapes HTTP /metrics endpoints on a schedule — pull model — discovering targets via service discovery like Kubernetes or Consul. Push is mainly for short-lived jobs through the Pushgateway.

# pull means Prometheus notices a dead target (failed scrape)
# apps just expose /metrics; SD keeps targets current as Pods churn

Counter is monotonic — requests_total. Gauge goes up and down — memory. Histogram buckets observations for quantiles. Summary does client-side quantiles. The type decides which queries are even valid.

http_requests_total                    # counter
node_memory_MemAvailable_bytes         # gauge
http_request_duration_seconds_bucket   # histogram

I'd divide the rate of 5xx-labeled requests by the rate of all requests over that window. rate() needs counters; sum() aggregates across instances.

sum(rate(http_requests_total{code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

I'd use histogram_quantile over the rate of _bucket series. You can't average pre-aggregated averages and get a correct tail latency.

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

I'd use recording rules to precompute expensive queries into new series for fast dashboards and reuse. Alerting rules evaluate conditions over time and fire to Alertmanager for grouping, routing, and silences.

- alert: HighErrorRate
  expr: job:error_rate:ratio5m > 0.02
  for: 10m
  labels: { severity: page }
  annotations: { summary: "5xx rate over 2%" }

I'd page on user symptoms and SLO burn — error ratio, latency — rather than raw CPU. Route causes to tickets or dashboards, require every page to be urgent and actionable, and delete or retune noise using pages-per-shift metrics.

Alert fatigue is a reliability bug. Cause alerts like CPU and disk are fine on dashboards and for secondary diagnosis; primary pages should be symptoms tied to SLOs. Multi-window burn-rate alerts page on fast burns and ticket on slow ones. Alertmanager grouping and inhibition collapse storms during known outages. I track MTTA/MTTR and pages that needed no action as hygiene KPIs. Symptoms over causes, severity routing, and continuous deletion of noise.

page:   SLO burn rate critical (5m+1h windows)
ticket: disk > 80% with 2d forecast to full
dashboard only: CPU steal, goroutine count

Interviewer often follows with: How would you introduce burn-rate alerts without double-paging during the migration?

You alert on how fast the SLO error budget is being consumed across short and long windows — paging when a fast burn would exhaust the budget soon, not on every small spike.

Classic multi-window multi-burn looks like 14.4× burn over 5m and 1h for a page, and 6× over 30m and 6h for a ticket. That catches both 'site is on fire now' and 'we're quietly chewing budget.' It lines alerts up with the same SLI used for the SLO, so dashboards, pages, and policy share one definition of reliability. Tune windows to your SLO period — 30d is typical — and pair with a sensible for: and Alertmanager routes.

# page if burning 2% budget in 1h (approx) AND still burning over 5m
# exact multipliers come from your SLO math / SRE workbook

Interviewer often follows with: What breaks if your SLI ignores a regional dependency that users feel?

It's following one request across services as a trace of spans with parent/child links, propagated via context headers like W3C traceparent. It shows where latency and errors sit in the call graph.

# waterfall shows 400ms in payments client, not in the gateway
# logs with the same trace_id explain the timeout

It's a vendor-neutral API, SDK, and collector for metrics, logs, and traces. You instrument once and export through a Collector — swapping backends becomes config, not a rewrite.

app SDK → OTLP → Collector (batch, sample, redact) → Jaeger/Tempo/vendor

I'd thread a shared trace_id through structured logs and spans, and use exemplars on latency metrics when they're available. Spike on a graph → exemplar or trace → logs for that id.

every log line: service, level, trace_id, span_id
every outbound call: inject traceparent

I'd use head sampling for cheap baseline traces and tail sampling to keep errors and slow outliers after the trace completes. Drop boring fast successes — never drop all 5xx just to save money.

Head sampling decides at the root span — cheap, but it may discard the one broken request. Tail sampling buffers spans in a Collector and decides with full knowledge of status, latency, and attributes, keeping the high-value traces. Cost controls: attribute allow-lists, span limits, and separate retention for raw vs aggregated. Guaranteed throughput for error traces is a reliability feature. Document sampling so on-call knows a missing trace may be intentional.

keep: errors, latency > SLO threshold, canary traffic
sample: 1% of successful GET /health-like traffic
drop: high-cardinality custom attributes on spans

Interviewer often follows with: How does sampling interact with exemplars on Prometheus histograms?

I'd drop or sample debug and noisy sources, enforce structured fields with allow-lists, hash or omit unbounded keys, route debug to short retention, and fix chatty apps at the source. Cardinality in labels and indexes is as dangerous as raw GB/day.

Cost drivers are bytes ingested, indexed fields, and retention. Practical moves: default level=info in prod, sample successful access logs, never index user-generated strings as high-cardinality labels, and use dynamic rate limits per service. Keep full fidelity for errors and security audits. Metrics shouldn't duplicate every log line — metrics for aggregates, logs for examples. Fix the emitter; don't just buy a bigger plan.

# Collector/processor: drop http.access if code=200 and sample 1%
# forbid labels: user_email, request_body
# retention: debug 3d, info 14d, audit 365d

Interviewer often follows with: When is sampling access logs unacceptable for compliance?

I'd ban unbounded labels on metrics, aggregate at exporters, use recording rules and longer-retention downsampling via remote-write tiers, and push per-request detail to logs and traces instead.

A single user_id label can create millions of series and OOM Prometheus. Relabel configs can drop bad labels at scrape time as a seatbelt. Prefer histograms with bounded le buckets over per-instance summaries when you need global quantiles. On Kubernetes, be careful with pod/instance on high-fanout metrics — aggregate to deployment level where you can for SLOs. Capacity-plan series count, not only disk.

# prometheus scrape relabel: drop user_id if present
# SLO metrics: only {service, route, code_class}

Interviewer often follows with: How would you detect a cardinality explosion before Prometheus falls over?

The short version is an SLI is what you measure — say success ratio — an SLO is the target like 99.9%, and the error budget is 1 minus SLO: the unreliability you're allowed to spend on change before you have to stabilize.

SLO: 99.9% success over 30d
budget: 0.1% ≈ 43 minutes of downtime / 30d
budget spent → feature freeze / reliability work

I'd pick something close to user experience: successful requests within a latency threshold — availability times freshness — not CPU. Count only user-facing critical routes, exclude synthetic health checks, and document exclusions.

Bad SLIs like CPU or pod restarts disconnect reliability policy from users. Good patterns: ratio of requests with code!~"5.." and latency under 300ms on checkout paths, or successful job completion for async workers. Separate SLOs per customer-critical journey if you need to. The window — 30d rolling vs calendar — changes burn math. Publish the PromQL next to the SLO so alerts and reports can't drift. Review quarterly; product changes invalidate SLIs.

slI = successful_fast / total_valid
successful_fast: code=~"2.." and latency_bucket <= 0.3s
exclude: route="/healthz", synthetic=true

Interviewer often follows with: How would you handle multi-endpoint services where one rare RPC is business-critical?

I'd detect and declare early, name an incident commander, mitigate first — rollback, shed load, failover — before deep root cause, communicate on a cadence, then resolve and run a blameless postmortem with owned actions.

Roles: IC decides, ops mitigates, comms handles customers and stakeholders, scribe keeps the timeline. Mitigation beats investigation while budget burns. Use feature flags and known rollback paths. Severity defines response time and audience. Afterward: restore SLO tracking, revoke emergency privileges, and open the postmortem within days. Calm structure under pressure beats hero debugging.

1) declare SEV and IC in channel
2) graph golden signals + last deploy
3) rollback / disable flag if strongly correlated
4) status update even if cause unknown

Interviewer often follows with: When do you stop rolling back and instead forward-fix?

Facts over fault: timeline, user impact, contributing factors, what went well and poorly, and concrete action items with owners and due dates. The goal is system learning — not blaming the human who touched the keyboard.

impact (users, duration, SLO burn)
timeline (UTC)
contributing factors (not "root cause" alone)
action items: owner + date + tracked ticket

I'd trigger the agreed policy: freeze or slow feature launches, prioritize reliability work that restores the SLI, and review whether the SLO or the implementation is wrong. The budget is a negotiation tool, not a surprise punishment.

Mature orgs write the policy before the crisis: what happens at 50% and 100% budget burn, who decides exceptions, and how customer commitments interact. Exhaustion should redirect engineering capacity to toil reduction, dependency hardening, and alert/SLO quality — not endless firefighting without systemic fixes. Sometimes the SLO is too tight for the architecture; renegotiate explicitly with data. Exceptions for must-ship legal changes need compensating risk acceptance. Organizational design matters as much as PromQL.

at 50% burn: reliability backlog gets priority slots
at 100%: feature freeze except P0 security/legal
exception: VP product + SRE approve with expiry

Interviewer often follows with: How would you stop teams from gaming the SLI by shrinking what they measure?

RED metrics with histograms, structured logs with trace_id, OTel traces around outbound calls, a golden dashboard, SLO plus burn alerts, and cardinality guardrails — shipped as a platform template so every service starts the same.

Platform engineering wins here: auto-instrument HTTP/gRPC, a standard label vocabulary like service/route/code, log/metric/trace correlation baked in, and a service scaffold with /metrics, health probes, and dashboard JSON. Define which alerts are mandatory for production readiness. Don't let each team invent label schemes. Cost: default sampling and log levels. Readiness review checks: SLI defined, runbook linked from the alert, and on-call can find traces in under two minutes.

[ ] RED dashboard
[ ] p99 + error-ratio burn alerts
[ ] OTel trace on egress
[ ] runbook URL on alert
[ ] no unbounded metric labels

Interviewer often follows with: What do you require before the service gets a public DNS name?

It groups related alerts, deduplicates, routes by labels to page vs ticket channels, and supports silences and inhibition so a parent outage doesn't spam every child alert.

severity=page  → PagerDuty
severity=ticket → Jira/Slack
inhibit: InstanceDown suppresses HighLatency on same instance

Synthetics give controlled probes — uptime, multi-region path checks — even when traffic is low. Real-user and SLI metrics capture actual experience including client diversity. I alert on both: synthetics for 'is it up,' RUM/SLI for 'is it good for users.'

Synthetics can miss issues that only hit certain tenants, devices, or payloads; RUM can be delayed or sparse at night. Best practice for me: critical-path synthetics from multiple regions with auth where needed, plus SLIs from production traffic. I don't let a green synthetic dashboard override a burning user SLO. Keep synthetics out of the SLI denominator — or label and exclude them — so you don't inflate availability.

synthetic: every 60s HTTPS GET /login from 3 regions → page on fail
SLI: real checkout success_fast ratio → burn alerts

Interviewer often follows with: How would you authenticate synthetics without creating a backdoor?

Toil is manual, repetitive, automatable operational work that scales with service size. I'd track it, cap on-call toil, and invest engineering time to knock out the top sources — pagers that need human babysitting are a smell.

toil: hand-restarting pods, manual certificate renewals
anti-toil: HPA, cert-manager, runbooks that are actually scripts

I'd use for: pending windows, rate over sufficiently long ranges, Alertmanager group_wait, and hysteresis via burn-rate or separate raise/clear thresholds — never alert on a single noisy scrape.

- alert: HighErrorRate
  expr: job:error_rate:ratio5m > 0.02
  for: 10m

If buckets skip your SLO threshold — say SLO is 300ms but buckets sit at 100ms and 500ms — histogram_quantile and success ratios get inaccurate. I'd align buckets with SLO boundaries and expected tails.

Exemplars and quantiles interpolate within buckets; wide gaps around the SLO create optimistic or pessimistic error. Redesign instrumentation with explicit boundaries at the SLO and common percentiles. Releasing a bucket change creates a series break — document it. This is a frequent production footgun in interviews.

# SLO: 300ms — include le="0.3"
buckets: [0.05, 0.1, 0.3, 0.5, 1, 2.5]

Interviewer often follows with: Do you need the same buckets on every service?

A runbook is the actionable guide for a specific alert: symptoms, dashboards, mitigation steps, escalation. Every page-worthy alert should link to one — alerts without runbooks train people to ignore pages.

annotations:
  summary: "Checkout error budget burning"
  runbook_url: https://wiki/runbooks/checkout-burn

I'd suspect a wrong SLI — averages hiding tails — missing regional labels, client-side failures not in server metrics, or sampling bias. Check p99, per-region RED, synthetics, traces for errors, and RUM or app-store crash data.

Classic traps: success ratio excludes timeouts classified as client disconnects; load balancer 5xx not counted on app metrics; one shard failing behind a healthy average; CDN cache masking origin death for some paths. Expand telemetry scope before declaring user error. Chase the user's path, not the convenient dashboard.

1) p99 vs avg
2) break down by region/zone/version
3) ingress/LB codes vs app codes
4) client RUM / mobile crashlytics
5) recent deploys + feature flags

Interviewer often follows with: How can canary analysis false-pass on averages but fail users in one geo?

I'd watch utilization and saturation — queue depth, thread wait, disk IO credit — under real traffic, load-test to find cliffs, and set autoscaling on leading indicators, not only CPU after users already hurt.

# scale on: concurrent requests or queue depth
# not only: CPU > 90% after latency already burned SLO

MTTD is mean time to detect; MTTR is mean time to recover. Good alerts minimize MTTD for user-impacting issues without creating so much noise that MTTR suffers from fatigue and lost trust.

too few alerts → high MTTD
too many pages → high MTTR (ignored/slow)

Same three: metrics, logs, and traces. Metrics show trends and SLOs, logs give event detail, traces show request paths across services. You correlate all three during incidents.

metrics: is error rate up?
traces: which dependency is slow?
logs: what exception did instance X throw?

The short version is an SLI is the measured indicator — say success ratio — an SLO is the target we aim for, and an SLA is the external contractual commitment, usually looser than the internal SLO.

SLI: successful HTTP requests / total
SLO: 99.9% over 30 days
SLA: 99.5% with customer credits

I'd identify top metrics by series count, drop unbounded labels, shard or remote-write to a long-term store, shorten local retention, and fix exporters at the source. Cardinality first, hardware second.

I'd use TSDB status and cardinality explorers to find offenders — things like http_requests_total{user_id=...}. Relabel-drop as an emergency brake, then patch apps. Consider recording rules that aggregate away high-fanout labels for SLOs. Federate or use Thanos/Mimir/Cortex for scale, but don't remote-write garbage. Set series limits per tenant if you're multi-tenant. Methodical cardinality triage beats buying bigger VMs.

1) top-N metrics by series
2) scrape relabel drops
3) fix instrumentation PRs
4) remote write + retention tiering

Interviewer often follows with: How would you attribute cardinality to a single team in a shared cluster?

I'd slice SLIs by region and critical journey and set regional SLOs or weighted budgets so a small region can't burn unnoticed and a large region can't hide a small one's pain.

Global averages hide localized failure — classic SRE footgun. Design: per-region success ratio, per mobile/web clients, and multi-window burn alerts on each slice. Traffic weights belong in reporting, not in erasing user pain. Document which SLO gates release. Dimensionality of SLIs is an architecture choice.

# alert: burn rate on sli:http_success{region="ap-south-1"}
# dashboard: SLO triangles per region, not one global number

Interviewer often follows with: How many SLO dimensions is too many for on-call to reason about?

I'd use tail-based sampling that keeps errors and slow traces at high rate, head-sample the rest low, and make sure exemplars link metrics to retained traces.

Head-only low sampling drops rare failures. Tail sampling in a gateway or collector inspects completed traces and retains on error/latency rules. Pair with exemplar support in Prometheus/Grafana. Budget: cap retained traces per day per service. A purposeful sampling policy beats 1% everywhere.

# OTel collector: sample 100% if status=ERROR or duration>2s
# else 1% probabilistic

Interviewer often follows with: What bias does keep-all-errors introduce when a dependency flaps?

I'd drop or sample debug in prod, index only high-value fields, route bulk logs to cold storage, keep structured events for errors, and measure logs needed per incident — not GB ingested.

Cost levers: field cardinality, debug-chatty libs, duplicate access logs at edge and app, and indefinite hot retention. Plan: parse once, tier storage — hot 7d / cold 90d — dynamic severity, and event-shaped logs over plaintext dumps. Keep audit and security logs complete. Cost is an SLO for the observability platform itself.

# hot: error+audit 7d searchable
# warm: info sampled 14d
# cold: raw object storage 90d, rehydrate on demand

Interviewer often follows with: Which log fields would you never drop for security investigations?

I'd scale on saturation or queue metrics — or a latency SLI — that lead the incident, not only CPU, and expose those metrics from the app runtime.

CPU can be low while request queues grow — blocked threads, connection pools. USE/RED: watch pool utilization, queue depth, and p99. Custom metrics for HPA or KEDA. Load-test to find the right signal. Leading indicators tied to the failure mode beat CPU after users already hurt.

# metric: app_queue_depth > 50 → scale out
# verify: CPU may stay at 20% while latency climbs

Interviewer often follows with: How would you prevent flapping when scaling on noisy queue metrics?

I'd emit deploy events as annotations or changelog markers, label metrics with version, retain traces and logs for the change window, and store a durable audit of who promoted which digest.

Change intelligence: deploy markers in Grafana, version labels on RED metrics, CI identity in provenance, and immutable artifact digests. Incident reviews should pull the marker timeline automatically. Retention must meet audit windows. Observability plus CI identity plus change calendar as one system.

http_requests_total{version="sha-abc"}
# Grafana annotation: deploy sha-abc by oidc:alice at T

Interviewer often follows with: How would you attribute a canary's SLO burn to a specific digest under shared pods?

I'd measure pages-per-shift and actionability, kill or downgrade always-firing alerts, route by symptom and SLO burn, and require every page to have a runbook link before it stays paging.

Fatigue is a reliability bug. Inventory alerts by fire rate and percent leading to action; delete flappers; convert infra noise to tickets; keep paging for user impact — SLO burn, error spikes. Every alert needs a team and a doc. Review weekly until pages per night is human. Celebrating silenced noise is as important as new dashboards.

# this week:
# 1) top 20 alerts by volume
# 2) delete/ack-rate >90% with no ticket → demote
# 3) every paging alert must link runbook URL
# KPI: pages/on-call-night < 5 actionable

Interviewer often follows with: What's the difference between a ticket alert and a page?

I'd emergency-drop the high-cardinality labels via scrape relabel, restore TSDB health, then patch instrumentation and add cardinality gates in CI and review.

Unbounded labels create a series per combination and can OOM Prometheus. Mitigate with metric_relabel_configs drop/keep, restart if corrupted, and find the offender via cardinality explorers or tsdb status. Fix apps to put IDs on logs and traces. Prevent with lint rules and per-tenant series limits. Stabilize the platform before blaming 'just add RAM.'

metric_relabel_configs:
  - regex: "user_id|request_id"
    action: labeldrop
# then: PR removing those labels from the exporter

Interviewer often follows with: Why can recording rules make a cardinality problem worse if written carelessly?

I'd check window length vs outage duration, the SLI definition — success vs availability — traffic weighting, and whether the burn threshold is too loose for short complete failures.

Classic miss: long windows dilute a short 100% outage; or the SLI counts synthetic checks that still pass while users fail; or the budget is so large a total outage barely burns. Fix: fast and slow burn pairs — say 1h/6h and 6h/3d — SLIs that match user journeys, and alert on absolute error spikes as a backup. Validate with historical incident replay. Distrust a green burn graph during known pain.

# did 30m at 0% success burn enough of the 30d budget to trip 2% / 1h?
# add: page on success ratio < 99% for 5m OR fast burn
# SLI: browser RUM or gateway 5xx — not only /healthz

Interviewer often follows with: When would a pure error-budget burn alert be the wrong primary page?

I'd move to tail-based sampling that keeps errors and high-latency traces at high rate, keep a low probabilistic sample for the happy path, and link metrics exemplars to retained traces.

Head sampling randomly drops the rare bad requests you need. Tail sampling in the collector inspects completed traces and retains on status/latency rules with a daily budget. Pair with structured logs carrying trace_id for the rest. Cost control: per-service caps and defensive drop of chatty health checks. Purposeful sampling beats uniform 1%.

# keep 100% if error OR p99-class latency
# else 1% probabilistic
# budget: max N spans/day/service

Interviewer often follows with: What bias appears if you keep 100% of errors during a dependency flap?

I'd stop the leak — scrub or redact at source or ingest — rotate exposed sessions and tokens, restrict who can query historical logs, and add CI/lint rules that block PII fields going forward.

Logs are a data store under compliance regimes. Immediate: deploy redaction processors, delete or quarantine hot indexes if policy requires it, rotate credentials, and notify privacy/security. Lasting: field allowlists, structured logging standards, and DLP-ish scanners in CI. Never 'we'll clean it next sprint' while tokens remain valid.

# collector: replace email/token fields with hash or drop
# app: log user_id (internal) not email; never Authorization headers
# CI: fail on patterns Authorization|password|ssn

Interviewer often follows with: Which is safer for correlation — hashing emails in logs or using opaque internal user IDs?

I'd look at user-journey SLIs — checkout success, RUM, synthetics — not host CPU panels. Then traces and logs for the failing step, and ask which dependency the dashboards don't cover.

Green host metrics with broken UX means you instrumented the wrong layer, or health checks that don't exercise checkout. Path: confirm from the edge — CDN/gateway status, synthetic checkout — identify the failing span, check canaries and feature flags, and only then dive into USE resource graphs. Fix the dashboard set to include journey RED. Dashboards lie when SLIs are vanity.

1) synthetic checkout / RUM conversion
2) gateway 5xx + latency on /checkout/*
3) trace: payment span errors
4) only then: CPU/disk of payment pods

Interviewer often follows with: Give an example of a health check that stays 200 while checkout is down.

Tonight I'd stabilize with first principles — RED, recent deploys, dependencies. Tomorrow I'd block paging alerts without runbooks and write the missing doc from the incident timeline.

Missing runbooks extend MTTR. Night-of: declare IC, capture timeline, use standard playbooks — rollback, shed load, failover. Next day: every alert rule requires annotations.runbook_url; PR checks reject pages without it; improve the doc with the commands that actually worked. Game days keep runbooks honest. An alert without a next step is unfinished work.

annotations:
  summary: "Checkout error budget fast burn"
  runbook_url: "https://runbooks/checkout-burn"
  dashboard_url: "https://grafana/.../checkout"

Interviewer often follows with: What belongs in a runbook vs a full architecture doc?

I'd recalibrate SLI quality and thresholds with incident replay, separate ticket vs page burns, show precision metrics, and only then restore paging — with a named owner for the SLO.

False SEVs usually mean bad SLIs — healthz, low-traffic noise — or windows that flap. Work: replay last month's tickets against proposed alerts; raise continuity requirements; exclude known maintenance; document when to page vs ticket. Publish alert precision weekly. Deleting SLO paging without replacement returns you to CPU roulette. This is socio-technical repair of the SLO program.

# for each past incident: would fast/slow burn have fired? T/F positive?
# target: >80% precision before re-enabling page
# owner: sre-checkout@

Interviewer often follows with: How would you handle a correct burn alert that the business still considers a false SEV?

I'd check remote-write queues, shard limits, and relabel drops on the write path — and treat missing samples as an observability SEV because under-reporting hides burns.

Silent remote-write failure is dangerous optimism. Symptoms: queue capacity alerts, 4xx/5xx from the backend, hashmod imbalance, or overly aggressive write relabel. Compare edge gateway success rates to stored SLI rates. Mitigate: buffer, spill to local, page on write failure, and never compute compliance-only SLOs solely from a lossy store. Telemetry pipelines need SLOs too.

prometheus_remote_storage_samples_failed_total
prometheus_remote_storage_queue_highest_timestamp_seconds
# compare: edge success ratio vs recording rule in long-term store

Interviewer often follows with: Why might a local Prometheus graph disagree with the global Thanos/Mimir view during a write outage?

I'd slice RED and SLIs by version or canary label and sticky pool, alert on per-version error rates, and verify cutover with traffic percent and unique version counts — not only aggregate green graphs.

Aggregates hide split-brain deploys. Instrument version labels on metrics, show dual burn charts during cutover, and run synthetic checks against both pools. Confirm Service/Gateway weights and session affinity behavior. Rollback criteria must include 'any version still serving with elevated errors,' not global averages.

http_requests_total{version="blue|green",code=~"5.."}
# alert: burn(version=green) OR (traffic_green>95% AND blue_errors high sticky remnant)

Interviewer often follows with: What label cardinality trade-off do you accept to get per-version SLIs?

I'd normalize on trace_id and exemplars, check NTP and clock skew between nodes and SaaS, and treat one signal as primary for time while using others for detail — documenting skew in the timeline.

Skew and different batching windows create ghost causality. Practices: require trace_id in logs, exemplars from metrics to traces, understand scrape/timestamp configs, and note processing delay in log pipelines. For IR, prefer gateway or RUM timestamps as user-truth. Fix chronic skew — NTP — as a platform item. Correlation is a designed contract, not hope.

log: {"trace_id":"...","ts":"..."}
metric exemplar → trace
# IR note: log ingest delay ~45s; prefer gateway ts

Interviewer often follows with: How would you detect systematic clock skew across a Kubernetes node pool?

I'd ship tiered runbooks, a single 'start here' dashboard with journey SLIs, auto-links from alerts, and a shadow/onboarding rotation — paging alerts without those artifacts don't ship.

Human reliability is part of the system. Design: alert → dashboard → runbook → rollback one-liner; dependency map; clear escalation. Keep runbooks short and command-oriented. Practice with game days. Reduce tribal knowledge by encoding it in alert annotations. MTTR is a product of docs and design, not heroics.

alert annotations → grafana journey board → runbook
runbook: 1) confirm user impact 2) last deploy 3) rollback 4) escalate
game day: quarterly checkout burn drill

Interviewer often follows with: What's the failure mode of a 40-page runbook during a SEV-1?