Observability & gates

Monitor/App Insights driving deploy decisions.

An airliner is only cleared to land itself in thick fog when its instruments cross-check and agree. No valid readings, no autoland, however confident the pilot feels. A deployment gate holds software to the same rule: the pipeline promotes a build only when live telemetry says the system is healthy. Azure Monitor is the instrument panel, covering metrics, logs and distributed traces. Azure DevOps supplies the gate that reads that panel before, during and after a deployment. Wire the two together and a release stops being an act of faith. The pipeline measures first. Then it promotes, or it refuses and pages someone.

Metrics, logs and traces: your three instrument feeds

A metric is a single number sampled over and over on a timer, the way a fuel gauge is read: CPU (central processing unit) percentage, Http5xx count, average response time. Metrics are cheap to store and quick to query, usually readable a minute or two after the event, and they are what alert rules evaluate. A log is the written record of one thing that happened: this request, this URL, this status code, this duration. Logs land in a Log Analytics workspace and you read them with KQL (Kusto Query Language, Microsoft's read-only language for querying telemetry). A trace staples together the log events from several services so you get the story of one request end to end, and you can see which hop added the 900 ms. Azure Monitor is the umbrella over all three. Application Insights is its application layer, collecting requests, dependencies and exceptions with little or no code change.

One internals detail catches almost everyone. Classic Application Insights retired in early 2024, so every App Insights resource is now *workspace-based*: the telemetry physically lands in Log Analytics tables. The App Insights query API still answers to the old table names (requests, exceptions). Query the workspace directly and those same rows are called AppRequests and AppExceptions. Same data, two names, and each door only opens for one of them. Point a gate at the wrong table and it gets zero rows back, reads zero as "nothing is broken", and waves the build through. A gate whose failure mode is *pass* is worse than having no gate, because you believe it.

# Workspace-based App Insights: one Log Analytics workspace per environment
az monitor log-analytics workspace create \
  --resource-group rg-prod --workspace-name law-prod --location westeurope \
  --query "{name:name, sku:sku.name, retentionDays:retentionInDays}"
# {
#   "name": "law-prod",
#   "retentionDays": 30,
#   "sku": "PerGB2018"
# }
 
# App Insights component wired to that workspace (CLI extension: application-insights)
az monitor app-insights component create \
  --app appi-shop --resource-group rg-prod --location westeurope \
  --workspace law-prod --application-type web \
  --query connectionString -o tsv
# InstrumentationKey=6b1f22e3-9a41-4c8e-b7d2-0f3c9e5a1d44;IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/;LiveEndpoint=https://westeurope.livediagnostics.monitor.azure.com/
 
# Codeless attach for App Service: the platform agent reads these two settings
az webapp config appsettings set --name app-shop --resource-group rg-prod \
  --settings APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=6b1f22e3-...;IngestionEndpoint=..." \
             ApplicationInsightsAgent_EXTENSION_VERSION="~3"
 
# Prove telemetry is flowing (classic table names through this API)
az monitor app-insights query --app appi-shop -g rg-prod \
  --analytics-query "requests | where timestamp > ago(5m) | summarize n=count(), p95=percentile(duration, 95)" \
  --query "tables[0].rows" -o json
# [
#   [
#     412,
#     187.4
#   ]
# ]

Alerts and action groups, written down as code

An alert rule is a smoke detector on a schedule. It checks a condition every so often (evaluationFrequency), looking back over a fixed window (windowSize), and when the condition holds it sets off an action group: a reusable who-to-tell list of email addresses, SMS numbers, webhooks or Azure Functions. That last one is where self-healing automation starts. Alerts built by clicking around the portal drift apart. Production ends up with the tuned thresholds, staging has none, and nobody can tell you why the limit is 10. Define them in Bicep or Terraform, right next to the app they watch. Every environment then gets identical alerting, and every threshold change arrives as a reviewed pull request.

param location string = resourceGroup().location
param siteId string
 
resource oncall 'Microsoft.Insights/actionGroups@2023-01-01' = {
  name: 'ag-oncall'
  location: 'global'
  properties: {
    groupShortName: 'oncall'
    enabled: true
    emailReceivers: [
      { name: 'ops', emailAddress: '[email protected]', useCommonAlertSchema: true }
    ]
  }
}
 
resource http5xx 'Microsoft.Insights/metricAlerts@2018-03-01' = {
  name: 'alert-http5xx-spike'
  location: 'global'
  properties: {
    severity: 1
    enabled: true
    scopes: [siteId]
    evaluationFrequency: 'PT1M'
    windowSize: 'PT5M'
    autoMitigate: true
    criteria: {
      'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria'
      allOf: [
        {
          criterionType: 'StaticThresholdCriterion'
          name: 'http5xx'
          metricName: 'Http5xx'
          operator: 'GreaterThan'
          threshold: 10
          timeAggregation: 'Total'
        }
      ]
    }
    actions: [ { actionGroupId: oncall.id } ]
  }
}
 
// Deploy (idempotent — rerun it on every pipeline run):
//   az deployment group create -g rg-prod -f monitor.bicep \
//     -p siteId=$(az webapp show -n app-shop -g rg-prod --query id -o tsv) \
//     --query properties.provisioningState
//   "Succeeded"

# Same alert on the Terraform track (action group mirrors the Bicep one)
resource "azurerm_monitor_metric_alert" "response_time" {
  name                = "alert-response-time"
  resource_group_name = azurerm_resource_group.prod.name
  scopes              = [azurerm_linux_web_app.shop.id]
  description         = "Avg response time above 1.5s for 5 minutes"
  severity            = 2
  frequency           = "PT1M"
  window_size         = "PT5M"
 
  criteria {
    metric_namespace = "Microsoft.Web/sites"
    metric_name      = "HttpResponseTime"
    aggregation      = "Average"
    operator         = "GreaterThan"
    threshold        = 1.5
  }
 
  action {
    action_group_id = azurerm_monitor_action_group.oncall.id
  }
}
 
# $ terraform apply -auto-approve
# azurerm_monitor_metric_alert.response_time: Creating...
# azurerm_monitor_metric_alert.response_time: Creation complete after 4s
# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Both tools end up creating the same Microsoft.Insights resources through ARM (Azure Resource Manager, the layer that actually builds things in Azure), so pick whichever one your infrastructure-as-code pipelines already standardized on. Severity runs from 0 (critical) down to 4 (verbose). Reserve paging for 0 and 1. autoMitigate: true (Terraform spells it auto_mitigate, and it is on by default) closes the alert by itself once the condition clears, which keeps the queue honest.

Gates: the pipeline asks before it promotes

A gate is a health question the pipeline has to hear a *yes* to before it moves on. Old-style release pipelines called them exactly that, with built-in options such as *Query Azure Monitor alerts* and *Invoke REST API*, re-run over and over between the approval and the deployment. In YAML (the indented text format Azure Pipelines uses for pipeline definitions) the same idea is called a check, and checks hang off an environment. A deployment job aimed at prod cannot start until every check on prod passes, and the built-in *Query Azure Monitor alerts* check keeps re-evaluating on an interval until no matching alert is active, or until it times out. That covers the moment *before* you deploy. Judging the deployment itself takes a bake gate: put the build somewhere safe, let traffic hit it for a while, query the telemetry, and fail the job if the numbers moved the wrong way.

# Stage starts only after every check on the 'prod' environment passes
# (attach "Query Azure Monitor alerts" under Pipelines > Environments > prod > Approvals and checks)
- stage: prod
  dependsOn: build
  jobs:
  - deployment: deploy_shop
    environment: prod
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureWebApp@1
            displayName: Deploy to staging slot
            inputs:
              azureSubscription: sc-prod   # workload identity federation, not a PAT
              appType: webAppLinux
              appName: app-shop
              resourceGroupName: rg-prod
              deployToSlotOrASE: true
              slotName: staging
              package: $(Pipeline.Workspace)/drop/shop.zip
 
          - task: AzureCLI@2
            displayName: 'Bake gate: no exception spike in 10 min'
            inputs:
              azureSubscription: sc-prod
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                sleep 600   # soak the slot under smoke + synthetic traffic
                N=$(az monitor app-insights query --app appi-shop-staging -g rg-prod \
                  --analytics-query "exceptions | where timestamp > ago(10m) | count" \
                  --query "tables[0].rows[0][0]" -o tsv)
                echo "exceptions in bake window: $N"
                if [ "$N" -gt 5 ]; then
                  echo "##vso[task.logissue type=error]Bake failed: $N exceptions in staging"
                  exit 1
                fi
 
          - task: AzureCLI@2
            displayName: Swap staging -> production
            inputs:
              azureSubscription: sc-prod
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: >
                az webapp deployment slot swap -g rg-prod -n app-shop
                --slot staging --target-slot production

# What a caught regression looks like in the run log (stage: prod):
#   Bake gate: no exception spike in 10 min
#     exceptions in bake window: 23
#     ##[error]Bake failed: 23 exceptions in staging
#   Stage prod: failed - swap step never ran; production untouched.
 
# Same story from the CLI (extension: azure-devops)
az pipelines runs list --org https://dev.azure.com/contoso --project shop \
  --top 2 --query "[].{run:buildNumber, result:result, branch:sourceBranch}" -o table
# Run         Result     Branch
# ----------  ---------  ---------------
# 20260114.2  succeeded  refs/heads/main
# 20260114.1  failed     refs/heads/main

Read that failed run from the bottom up. The artifact went to the *staging slot*. It soaked for ten minutes while the KQL gate counted fresh exceptions. The count came back 23, well over the threshold of 5, so the step exited non-zero and the stage failed. The swap step never ran, so production never saw the build. Rolling back here means nothing more than declining to promote. Watch the scoping, too. This gate queries the staging slot's own App Insights resource. If your team shares one resource across several services, filter with | where cloud_RoleName == 'app-shop-staging', or your release gets blocked by somebody else's bug.

Slot swaps: sticky settings and warm-up

A deployment slot is a second, complete copy of your App Service app with its own hostname and its own configuration, like a spare stage on a theatre's revolving floor. A swap turns the floor so the other stage faces the audience, which in Azure terms means changing which slot answers the production hostname. The turn is deliberately slow and warm. Azure first copies production's configuration onto the staging slot, restarts its worker processes, then fires warm-up requests at the path named in WEBSITE_SWAP_WARMUP_PING_PATH (the site root / unless you say otherwise), counting only the HTTP status codes listed in WEBSITE_SWAP_WARMUP_PING_STATUSES as success (every code counts as success until you set that one). Routing flips only once warm-up succeeds, so live traffic never lands on a cold process. App settings follow the code by default. Settings marked sticky (slot settings) stay behind with the slot name instead. A handful are sticky whether you ask or not: custom domains, non-public certificates and TLS (Transport Layer Security) bindings, IP (Internet Protocol) address restrictions, and scale settings. Make your Application Insights connection string sticky as well. Leave it unsticky and it rides along with the code on every swap, so the instant you swap, production starts reporting into the staging resource and staging reports into production. Every gate and alert you built above is now reading the wrong app.

# Create the staging slot, cloning production's current config
az webapp deployment slot create -g rg-prod -n app-shop \
  --slot staging --configuration-source app-shop
 
# --slot-settings marks settings STICKY: they stay with the slot on swap.
# Each slot keeps its own telemetry stream, so the bake gate reads staging in isolation.
az webapp config appsettings set -g rg-prod -n app-shop --slot staging \
  --slot-settings APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=0f9d...staging..."
# [
#   ...
#   {
#     "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
#     "slotSetting": true,
#     "value": "InstrumentationKey=0f9d...staging..."
#   }
# ]
 
# Gate the swap itself on a real health endpoint (WEBSITE_* platform settings,
# not sticky - they travel with whichever config is being warmed up)
az webapp config appsettings set -g rg-prod -n app-shop \
  --settings WEBSITE_SWAP_WARMUP_PING_PATH="/healthz" \
             WEBSITE_SWAP_WARMUP_PING_STATUSES="200"
 
# Two-phase swap: preview applies PROD's sticky settings to staging first,
# so you validate the exact config production traffic is about to receive
az webapp deployment slot swap -g rg-prod -n app-shop --slot staging --action preview
# ...smoke-test https://app-shop-staging.azurewebsites.net, then complete (or --action reset):
az webapp deployment slot swap -g rg-prod -n app-shop \
  --slot staging --target-slot production --action swap
 
az webapp show -g rg-prod -n app-shop --query "{host:defaultHostName, state:state}"
# {
#   "host": "app-shop.azurewebsites.net",
#   "state": "Running"
# }

Limits, trade-offs, and what comes next

Learn the physics before you trust the instruments. Metric alerts usually see data within a minute or two, but log ingestion can lag several minutes, so a short bake can sail straight past a slow-burn regression. Size the window to your traffic: ten minutes on a quiet service might cover forty requests, which tells you nothing at all. Gates add lead time on purpose, so tune them per stage rather than setting one number everywhere. Log Analytics bills by the gigabyte ingested, which pushes high-volume services toward sampling. App Insights' adaptive sampling at least keeps every item of a sampled trace together, but sample hard enough and rare errors vanish from the data your gate is reading. And a gate is only as good as its query. Run your KQL against a time window you know was bad before you let it decide anything about production.

The working checklist. One Log Analytics workspace per environment. Alerts and action groups in the same infrastructure-as-code module as the app they watch. Gates on symptoms users feel (error rate, p95 latency meaning the slowest 5 percent of requests, availability) rather than on causes like CPU. Every KQL gate against a shared resource scoped by cloud_RoleName. Swap warm-up pointed at a real health endpoint instead of /. When a gate fails, that is the system doing its job. When a gate *passes* and production breaks anyway, the telemetry you wired up here becomes your evidence trail, and the next lesson turns those signals into incident response and blameless improvement.

Pick your thresholds from your own data, never from someone else's blog post. Before you write if [ "$N" -gt 5 ], run that same exception count over thirty days of ordinary traffic and see what normal looks like on your service. If a quiet Tuesday afternoon already produces four handled exceptions, a threshold of 5 fires on nothing, and you become the person who turned the gate off. Record the number you measured, and the date you measured it, in a comment beside the threshold. Traffic grows. The number goes stale. Whoever inherits the pipeline needs to know what it was based on.

Roll this out in order. One availability test, one error-rate alert, one dashboard the on-call engineer actually opens, and only then the gates. A gate nobody believes in gets overridden by hand on the first busy Friday. A gate tied to a number the team agreed on, such as the SLO burn rate, earns its keep. The cheapest useful version of everything above is a post-deploy job that queries App Insights for exceptions in the first fifteen minutes after a release, while rolling back is still cheap.

Try this

Create a metric alert on HTTP 5xx errors for a lab web app, then run the gate by hand before you automate it: check the metric yourself in the terminal before you approve a deployment to a prod environment. Doing it manually a few times teaches you what a normal reading looks like on your app, which is the number you will later hard-code. Bonus round: attach the real Azure Monitor gate under Approvals and checks, if your organization's SKU (the Azure DevOps pricing tier you are on) offers it.

az monitor metrics list --resource <webappResourceId> --metric Http5xx --interval PT1M -o table
az monitor app-insights component show -g rg-lab -a contoso-ai --query "{appId:appId,name:name}" -o json

$ az monitor metrics list --resource ... --metric Http5xx --interval PT1M -o table
Timestamp             Total
--------------------  -----
2026-07-24T01:00:00Z  0
2026-07-24T01:01:00Z  3
 
# Sample output — gate fails if 5xx exceeds threshold during canary window.

Takeaway

Let telemetry decide when a build moves, not the calendar and not anyone's gut feel. Healthy numbers promote, red numbers halt. And the gate only earns that authority if a broken query makes it fail rather than pass.

Next: build a service dashboard around the four golden signals (latency, traffic, errors, saturation) and wire your production environment check to one of them.

Related