Incident response & improvement

Blameless post-mortems; the DevOps loop.

Commercial flying didn't get safe by finding pilots who never make mistakes. It got safe because every crash and every near-miss gets taken apart by people whose job is to change the machine: add a line to the checklist, wire in a cockpit alarm, redesign the lever that two crews grabbed by accident. Punishing the crew was never the point. Incident response in DevOps copies that habit wholesale. An incident (any unplanned drop in the service your users actually touch) gets contained fast, and then a blameless post-mortem, a written analysis that names broken systems and never names people, turns the mess into permanent improvements to the way you ship.

On Azure every step of that loop has real machinery behind it. Azure Monitor notices the problem and wakes whoever is on call through an action group (a saved list of who to page and how to reach them). An App Service deployment slot swap puts production back on the old build in roughly two minutes. The run history in Azure Pipelines reconstructs exactly what changed and when. Azure Boards carries the follow-up work, and Bicep or Terraform turn the fixes into reviewed, versioned code instead of somebody's good intentions.

Contain first: the rollback is a swap

Production is broken, so diagnosis waits. Containment goes first, and on App Service your fastest containment is the same slot swap you used to deploy. A refresher on the internals, because rollback rests on them. A swap never moves code anywhere. App Service takes the target slot's configuration, applies it to the source slot's workers, restarts them, and warms them up by pinging the site root (or the path you set in WEBSITE_SWAP_WARMUP_PING_PATH) until the response code shows up in WEBSITE_SWAP_WARMUP_PING_STATUSES. Only then does it trade the routing rules between the two slots. Here is the consequence that matters at 3 a.m. After every swap, the build that used to be production is still running in the staging slot, warm and untouched. Rolling back means running the same swap a second time.

Configuration is where teams get caught out. By default, app settings and connection strings follow the *app* through a swap. Anything that differs per environment (the Key Vault address, the database connection string, APPLICATIONINSIGHTS_CONNECTION_STRING) has to be marked sticky, which the portal calls a 'deployment slot setting', so it stays welded to the *slot* instead. Miss that and your rollback quietly points production at the staging database. A few things are always sticky and never swap at all: managed identities, custom domains, TLS (Transport Layer Security, the certificate that puts the padlock on https) bindings, and scale settings.

# Mark per-environment settings STICKY so they never travel with a swap.
# --slot-settings (not --settings) pins them to the slot:
az webapp config appsettings set \
  --resource-group rg-checkout-prod --name checkout-api --slot staging \
  --slot-settings KEYVAULT_URI=https://kv-checkout-stg.vault.azure.net/
 
az webapp config appsettings list -g rg-checkout-prod -n checkout-api --slot staging
# [
#   { "name": "KEYVAULT_URI", "value": "https://kv-checkout-stg.vault.azure.net/",
#     "slotSetting": true },
#   { "name": "RELEASE_ID",   "value": "20260714.3",
#     "slotSetting": false }
# ]
 
# Deploy with a two-phase swap: apply prod config to staging, validate, complete.
az webapp deployment slot swap -g rg-checkout-prod -n checkout-api \
  --slot staging --target-slot production --action preview
# ...staging now runs prod config — smoke-test it, then:
az webapp deployment slot swap -g rg-checkout-prod -n checkout-api \
  --slot staging --target-slot production --action swap
 
# 09:47 — Http5xx alert fires. The old build is still in staging. Swap back:
az webapp deployment slot swap -g rg-checkout-prod -n checkout-api \
  --slot staging --target-slot production
# (no output on success — production is on the previous build ~2 minutes later)

Know what a swap-back does not undo. It reverses code and non-sticky configuration, and that is the whole list. If the bad release also ran a database migration, the migration is still applied, so the old code has to keep working against the new schema. That is precisely why the zero-downtime lesson insisted on backward-compatible, expand/contract migrations.

Triage: what changed?

With users out of danger, ask the question that starts every investigation: what changed? In a healthy Azure DevOps setup nothing reaches production except through a pipeline, so the answer is already written down in run history. Every run records the commit it built, the artifacts it published, who approved it and when it finished. The azure-devops extension for the az command-line tool puts all of that in your terminal, and tagging the suspect run leaves the post-mortem a permanent link to the evidence.

# Which run shipped the bad build? (requires: az extension add --name azure-devops)
az pipelines runs list --organization https://dev.azure.com/contoso \
  --project checkout --top 3 --output table
 
# Run ID  Number      Status     Result     Pipeline Name  Source Branch  Queued Time          Reason
# ------  ----------  ---------  ---------  -------------  -------------  -------------------  ------------
# 4812    20260714.3  completed  succeeded  checkout-ci    main           2026-07-14 09:41:22  individualCI
# 4811    20260714.2  completed  succeeded  checkout-ci    main           2026-07-14 08:03:47  individualCI
 
# Pin the evidence to the run so the post-mortem links straight to it:
az pipelines runs tag add --run-id 4812 --tags incident-1043 \
  --organization https://dev.azure.com/contoso --project checkout
# [
#   "incident-1043"
# ]

az pipelines runs show --id 4812 prints the whole record: what triggered the run, which commit it built, which artifacts it published. Its timeline puts the swap at 09:44, three minutes before the alert fired. Correlation is not proof of cause. It is still an excellent place to start digging.

Learn without blame

A post-mortem is a short document with four parts: a factual timeline built from pipeline runs and Azure Monitor data rather than anyone's memory, the impact on users, the contributing factors, and action items with an owner and a date on each one. Notice that factors is plural. Real incidents almost never have one root cause. A schema change got merged without a contract test, and review didn't flag it, and the deploy stage verified nothing after the swap, and the only alert anyone had built was watching CPU (central processing unit, the raw compute load) instead of errors. Four factors. Four fixes.

Blameless means the write-up names systems, not colleagues. Not 'Priya deployed a breaking change' but 'the pipeline let a breaking schema change reach production with nothing in the way to catch it'. That is an information strategy before it is a kindness. Hold the review inside 48 hours while people still remember the details, and file every action item where work actually gets scheduled, with a name on it:

az boards work-item create --type "Product Backlog Item" \
  --title "PM #1043: contract test for cart-service schema changes" \
  --assigned-to [email protected] \
  --organization https://dev.azure.com/contoso --project checkout \
  --output table
 
# ID    Type                  Title                                              Assigned To        State
# ----  --------------------  -------------------------------------------------  -----------------  -----
# 2311  Product Backlog Item  PM #1043: contract test for cart-service schema…   [email protected]  New

Codify the detection: alerts as code

The finding that shows up in more post-mortems than any other is 'nothing was alerting on the thing that broke'. Resist fixing that with clicks in the portal. A hand-built alert lives in exactly one environment, never shows up in a code review, and drifts or disappears without anyone noticing. Written in Bicep or Terraform, the same languages this course used for infrastructure, that alert gets reviewed like code, deploys identically to every environment, and comes back on its own if someone deletes it. Here is the alert that would have caught this incident at minute one instead of minute six, in both dialects:

param actionGroupId string
 
resource webApp 'Microsoft.Web/sites@2023-12-01' existing = {
  name: 'checkout-api'
}
 
resource http5xx 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'checkout-http5xx'
  location: 'global'
  properties: {
    severity: 1                       // Sev1 → pages on-call via the action group
    enabled: true                     // required property; alert is active once deployed
    scopes: [webApp.id]
    evaluationFrequency: 'PT1M'       // check every minute...
    windowSize: 'PT5M'                // ...over a rolling 5-minute window
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [{
        name: 'http5xx'
        metricName: 'Http5xx'
        operator: 'GreaterThan'
        threshold: 10
        timeAggregation: 'Total'
        criterionType: 'StaticThresholdCriterion'
      }]
    }
    actions: [{ actionGroupId: actionGroupId }]
  }
}
 
// az deployment group create -g rg-checkout-prod -f monitor.bicep \
//   --parameters actionGroupId=$AG_ID -o table
//
// Name     State      Timestamp                  Mode         ResourceGroup
// -------  ---------  -------------------------  -----------  ----------------
// monitor  Succeeded  2026-07-14T11:02:41+00:00  Incremental  rg-checkout-prod

resource "azurerm_monitor_metric_alert" "http5xx" {
  name                = "checkout-http5xx"
  resource_group_name = azurerm_resource_group.prod.name
  scopes              = [azurerm_linux_web_app.checkout.id]
  severity            = 1
  frequency           = "PT1M"
  window_size         = "PT5M"
 
  criteria {
    metric_namespace = "Microsoft.Web/sites"
    metric_name      = "Http5xx"
    aggregation      = "Total"
    operator         = "GreaterThan"
    threshold        = 10
  }
 
  action {
    action_group_id = azurerm_monitor_action_group.oncall.id
  }
}
 
# terraform apply
#   # azurerm_monitor_metric_alert.http5xx will be created
# Plan: 1 to add, 0 to change, 0 to destroy.
# azurerm_monitor_metric_alert.http5xx: Creating...
# azurerm_monitor_metric_alert.http5xx: Creation complete after 12s
# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

PT1M and PT5M are ISO-8601 durations (an international standard for writing dates and time spans as plain text), so the alert evaluates once a minute across a rolling five-minute window. Severity 1 is the level your action group is wired to page on. Pick one dialect per repository and stay with it. Deploying the same alert from Bicep and from Terraform leaves you with two tools holding two versions of the truth, each politely undoing the other's work.

Close the loop inside the pipeline

Second most common finding: nothing verified the deploy. Fix that with a stage, not with a habit. This stage runs the moment the swap finishes, and if the health probe comes back unhealthy it swaps production back without waiting for a human to read anything:

- stage: VerifyProd
  dependsOn: SwapToProd
  jobs:
  - job: smoke
    steps:
    - task: AzureCLI@2
      displayName: Smoke test, auto-rollback on failure
      inputs:
        azureSubscription: prod-svc-connection
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          code=$(curl -s -o /dev/null -w '%{http_code}' \
            https://checkout-api.azurewebsites.net/healthz)
          echo "healthz -> $code"
          if [ "$code" != "200" ]; then
            az webapp deployment slot swap -g rg-checkout-prod -n checkout-api \
              --slot staging --target-slot production   # swap back
            exit 1
          fi
 
# Run log for a bad deploy:
#   healthz -> 503
#   ##[error]Bash exited with code '1'.
#   Stage VerifyProd: Failed — production already swapped back automatically.

The trade-off is flapping. One transient network blip can roll back a release that was perfectly healthy. Production-grade versions retry the probe a few times, watch Application Insights failure *rates* rather than one endpoint, or reuse the alert-based deployment gates from the observability lesson. Start blunt, then tighten as your signals earn your trust.

One loop, fifteen lessons

Step back and the whole course turns out to be a single system. Branch policies in Azure Repos keep unreviewed code out. YAML (a plain-text format for writing configuration) pipelines and agent pools build every commit the same way, and Artifacts puts a version number on whatever you ship. Bicep and Terraform make the environments underneath disposable and repeatable, delivered through their own infrastructure-as-code pipelines. Release stages, approvals and progressive delivery walk each change toward users in small reversible steps, and the slot swap makes the final step boring. DevSecOps scanning throws out vulnerable dependencies and leaked secrets before they ship. Observability reports honestly on what happened afterwards. Incident response closes the ring: what production teaches you, through outages, metrics and the near-misses nobody had to page for, flows back in as new tests, stricter gates and better alerts.

That ring is why the strongest teams get speed *and* stability instead of trading one away for the other. Small, frequent deployments are individually low-risk and easy to reverse, and every failure leaves the next deployment a little safer. No single tool hands you that. The loop does. Build it, run it, and make every incident leave the pipeline better than it found it.

Detect, respond, learn, change the system. Azure hands you raw materials for all four: Monitor for detection, slots and pipelines for response, Boards for the follow-up, Bicep or Terraform for making a fix permanent. What Azure cannot supply is what happens in the room after the outage. Two teams with identical tooling produce either a list of guardrails or a list of names, and only one of them gets quieter on-call weeks.

So write down the factors, not the villains: the alert nobody built, the manual step someone forgot, the service with no clear owner. Turn each factor into a specific change with a specific home, an alert in Bicep, a gate in the YAML, a line in the runbook, an automated swap-back. Do that for a quarter and your mean time to restore (MTTR, the clock running from 'production is broken' to 'production is fine') drops. Not because anyone is trying harder. Because the path is paved.

Try this

Run a small game day. Break a lab app on purpose, let the action group page you, roll it back with a slot swap or by redeploying the previous pipeline run, then write a one-page blameless note and file three system fixes as Azure Boards work items.

az monitor action-group list -g rg-lab -o table
az boards work-item create --title "Add deploy gate on Http5xx" --type Task --project <project> 2>/dev/null || true
az webapp deployment slot swap -g rg-lab -n contoso-web --slot staging --target-slot production

$ az monitor action-group list -g rg-lab -o table
Name          Location
------------  --------
ag-oncall     global
 
# Sample output — Boards item 3184 created; slot swap restored previous version in ~2 minutes.

Takeaway

Remember: runbooks and a slot swap buy back the minutes, but the learning only sticks when the post-mortem is blameless and ends in artifacts you can point at, an alert in Bicep, a gate in the pipeline, a contract test in CI (continuous integration, the automated build that runs on every commit).

Next: put mean time to detect (MTTD, how long the outage ran before anyone knew) and mean time to restore on one plain dashboard, then make it a rule that no severity-1 incident closes until at least one preventive work item has been merged.

Related