Progressive delivery

Blue/green, canary, rolling, auto-rollback.

A city crew replacing a water main never shuts off the old pipe and hopes the new one holds. They lay the new pipe alongside the old one, open a valve to a few blocks, watch the pressure gauges, and cut the whole city over only once the readings stay boring. The old pipe stays pressurized the entire time, in case they need it back. Progressive delivery is that discipline applied to software. You expose a new version to real production traffic a slice at a time, measure its health while it runs, and keep a fast path back for the whole rollout.

The number that matters is blast radius: how many users a bad release hurts before you stop it. A plain deploy, where everything flips at once, has a blast radius of 100%. Three strategies shrink that number. Blue/green, canary and rolling each cut it a different way, and each charges you a different price for the safety. On Azure the parts are real and named: App Service *deployment slots*, Azure Pipelines *deployment jobs* with a canary strategy, and Azure Monitor alerts wired in as gates that can stop a rollout without asking anyone.

Blue/green, canary, rolling: what each one costs you

Blue/green runs two identical environments. Blue serves production while you deploy and test the new version on green. Cutover moves *all* traffic in one move, and rollback means pointing traffic back at blue, which is still running and still warm. That is the fastest rollback you can buy. The price: you pay for two environments during the overlap, and anything holding state has to work for both versions at the same time. A database schema change that green needs but blue cannot read has already broken your rollback path, before you ever try to use it.

Canary takes its name from the caged birds miners carried underground, a small sensitive thing that shows trouble before it reaches everyone. You release to a thin slice of real users first, 10%, then 25%, then everyone, widening only while the health numbers stay clean. Tightest blast radius of the three, and the most machinery to build, because you need traffic splitting plus telemetry (measurements streaming back out of the running app) that you actually trust. The whole strategy is watch the metrics and decide. On App Service, slot traffic routing gives you a canary with no service mesh at all: az webapp traffic-routing set --distribution staging=10 sends 10% of production traffic to the slot.

Rolling updates swap instances out in batches. Pull a few machines from the pool, update them, health-check them, put them back, repeat until none of the old version is left. There is no second environment on the bill, which is why it is the default on VM scale sets (groups of identical virtual machines Azure manages as one unit) and on Kubernetes. What you pay instead is the *mixed-version window*. Version N and version N+1 serve live traffic side by side for the entire rollout, so your APIs (the contracts other services call), your message formats and your database schemas all have to tolerate both at once. Rolling back means walking every batch back the way it came. Minutes, not seconds. In Azure Pipelines the rolling strategy targets VM resources, and maxParallel sets how many machines update at a time.

Blue/green on App Service: slots, sticky settings, and the swap

An App Service deployment slot is a second live copy of your app with its own hostname and its own configuration, running on the plan you already pay for (Standard, Premium or Isolated tier). The routine: push the new build to a staging slot, open the slot's own URL and check it, then swap. A swap does not copy files anywhere. App Service takes the production slot's settings, applies them to the staging workers, restarts those workers, warms them up by sending HTTP pings until they answer, and only then trades the routing between the two slots. Production never gets handed a cold worker. The old version keeps running in whichever slot you swapped it into, so your rollback is sitting right there, already warm.

Configuration is where swaps bite. By default, app settings and connection strings *travel with the app* through a swap. Settings you mark as slot settings (everyone calls them sticky) stay pinned to their slot instead. Get that backwards and you have a security incident, not a config bug. An unpinned DB_CONNECTION means production starts writing to your staging database the moment the swap lands, while the staging slot, which usually has looser access control and more people holding keys to it, inherits your production credentials. Pin environment identifiers, pin connection strings, pin anything with the word *staging* in its value. Two WEBSITE_* app settings control the warm-up: WEBSITE_SWAP_WARMUP_PING_PATH aims the ping at a real health endpoint instead of the default /, and WEBSITE_SWAP_WARMUP_PING_STATUSES lists which HTTP status codes count as warm. A few things never swap at all. Managed identities (the app's own Azure sign-in, with no password for anyone to steal), custom domains, TLS/SSL bindings (the certificates that make HTTPS work) and scale settings all stay with the slot. So code that fetches tokens through managed identity starts running under the *production* slot's identity, and production's permissions, the second it lands there.

# 1. Create a staging slot on the existing app (Standard tier or higher)
az webapp deployment slot create -g app-rg -n contoso-web --slot staging
 
# 2. Deploy the new build to the SLOT, never straight to production
az webapp deploy -g app-rg -n contoso-web --slot staging \
  --src-path ./app.zip --type zip
# { "active": true, "complete": true, "deployer": "OneDeploy",
#   "site_name": "contoso-web", "status": 4, ... }    <- 4 = success
 
# 3. Pin slot-specific settings so they DON'T travel during the swap
az webapp config appsettings set -g app-rg -n contoso-web --slot staging \
  --slot-settings ENVIRONMENT=staging DB_CONNECTION="Server=stg-sql;..."
# [ { "name": "ENVIRONMENT", "slotSetting": true, "value": "staging" }, ... ]
 
# 4. Rehearse: applies prod settings to staging workers; no traffic moves yet
az webapp deployment slot swap -g app-rg -n contoso-web \
  --slot staging --action preview
 
# 5. Verify https://contoso-web-staging.azurewebsites.net, then complete
az webapp deployment slot swap -g app-rg -n contoso-web \
  --slot staging --action swap       # exits 0, no output = swapped
 
# 6. Bad release? Swap again — instant rollback, old version is still warm
az webapp deployment slot swap -g app-rg -n contoso-web --slot staging

Canary in Azure Pipelines YAML

In pipeline YAML (the plain-text file that defines the pipeline), a deployment job points at an environment and takes a strategy: runOnce, rolling or canary. The canary strategy runs three hooks, deploy, routeTraffic and postRouteTraffic, once for every entry in increments, and hands each pass the current slice through $(strategy.increment). Then on: failure and on: success decide whether the release backs out or graduates. The health gate below is the *Query Azure Monitor alerts* task. It runs agentless, on a *server* job, which is why the hook carries pool: server: no build machine is involved, Azure DevOps calls Monitor itself. The task fails the step if any matching alert has fired, that failure trips on: failure, and the canary is torn down with nobody paged.

jobs:
- deployment: web_canary
  environment: prod.web           # environment 'prod', Kubernetes resource 'web'
  pool:
    vmImage: ubuntu-latest
  strategy:
    canary:
      increments: [10, 25]        # 10% of pods, then 25%, then full
      deploy:
        steps:
        - task: KubernetesManifest@1
          inputs:
            action: deploy
            strategy: canary
            percentage: $(strategy.increment)
            manifests: manifests/deployment.yml
      postRouteTraffic:           # the health gate between increments
        pool: server              # AzureMonitor runs agentless (server job)
        steps:
        - task: AzureMonitor@1    # "Query Azure Monitor alerts"
          inputs:
            connectedServiceNameARM: azure-prod
            ResourceGroupName: app-rg
            filterType: none      # any fired alert in the RG fails the gate
      on:
        failure:                  # gate tripped -> delete the canary pods
          steps:
          - task: KubernetesManifest@1
            inputs:
              action: reject
              strategy: canary
              manifests: manifests/deployment.yml
        success:                  # all increments clean -> becomes stable
          steps:
          - task: KubernetesManifest@1
            inputs:
              action: promote
              strategy: canary
              manifests: manifests/deployment.yml

# Queue the pipeline, then watch the canary walk its increments
az pipelines run --name web-canary --branch main -o table
# ID   Number      Status      Result   Pipeline Name  Source Branch  Reason
# ---  ----------  ----------  -------  -------------  -------------  ------
# 812  20260714.3  notStarted           web-canary     main           manual
 
# In the run log, each increment is its own gated phase:
#   Canary deploy (10%)   KubernetesManifest: baseline+canary pods up    OK
#   postRouteTraffic      AzureMonitor: 0 fired alerts                    OK
#   Canary deploy (25%)   KubernetesManifest: baseline+canary pods up    OK
#   postRouteTraffic      AzureMonitor: 1 fired alert (p95 > 800ms)   FAILED
#   on: failure           KubernetesManifest reject: canary pods deleted
# Result: partiallySucceeded — bad build never reached 100% of users

Slots and stickiness as code

Ticking a setting sticky in the portal is drift waiting to happen. The next environment somebody builds will not have that tick, and you will find out about it halfway through a swap. Declare the slots and their sticky settings in the same IaC (infrastructure as code, the files that create your cloud resources) that creates the app. In Bicep, Azure's own deployment language, sticky settings live in a slotConfigNames config resource hanging off the site. In Terraform's azurerm provider they are the sticky_settings block, and azurerm_web_app_active_slot makes the swap itself declarative: repoint its slot_id and the next apply performs the swap for you.

param planId string
 
resource web 'Microsoft.Web/sites@2024-04-01' existing = {
  name: 'contoso-web'
}
 
resource staging 'Microsoft.Web/sites/slots@2024-04-01' = {
  parent: web
  name: 'staging'
  location: resourceGroup().location
  properties: {
    serverFarmId: planId
    httpsOnly: true
  }
}
 
// Sticky settings, declared: these names never travel in a swap
resource sticky 'Microsoft.Web/sites/config@2024-04-01' = {
  parent: web
  name: 'slotConfigNames'
  properties: {
    appSettingNames: [ 'ENVIRONMENT', 'DB_CONNECTION' ]
  }
}
 
// az deployment group create -g app-rg -f main.bicep -p planId=$PLAN_ID
//   ..."provisioningState": "Succeeded"

resource "azurerm_linux_web_app" "web" {
  # ...name, service_plan_id, site_config...
  sticky_settings {
    app_setting_names = ["ENVIRONMENT", "DB_CONNECTION"]
  }
}
 
resource "azurerm_linux_web_app_slot" "staging" {
  name           = "staging"
  app_service_id = azurerm_linux_web_app.web.id
  site_config {}
  app_settings = { ENVIRONMENT = "staging" }
}
 
# Declarative blue/green: repointing slot_id performs a swap on apply
resource "azurerm_web_app_active_slot" "active" {
  slot_id = azurerm_linux_web_app_slot.staging.id
}
 
# terraform apply
#   azurerm_linux_web_app_slot.staging: Creation complete after 34s
#   azurerm_web_app_active_slot.active: Creation complete after 1m12s
#   Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Rollback is the point

Every strategy above is worth exactly as much as its reflexes. Progressive delivery pays off when the pipeline names its health signals *before* the deploy starts (error rate, p95 latency meaning the response time only the slowest 5% of requests go past, failed health probes, a specific alert rule by name) and reverts to the last good version by itself the moment those signals go bad. That is what turns a bad release from an outage into a blip. The canary hook rejects, or the slot swaps back, in seconds, usually before the first user has finished typing a support ticket. Feature flags from the branching lesson round out the kit: a bad *feature* switches off with no redeploy at all.

You own the strategy layer now: who gets the new version, in what order, and how it takes itself back. What we skated past is doing all of that without dropping a single request. Connection draining, how deep a warm-up really needs to go, what happens to session state across a swap, and the database migrations that decide whether a blue/green cutover is reversible at all. That is next: Zero-downtime deployment.

If you want a decision rule rather than three descriptions, pick by what you already have. A second environment you can afford for an hour, and a release that touches no schema? Blue/green, because the rollback is a traffic re-point. No trustworthy per-version metrics coming out of the app yet? A canary is theater, so run rolling with small batches and a real health probe. Genuinely risky release, and Application Insights already breaks error rate down by version? Canary earns its extra moving parts.

Rehearse the rollback on a day when nothing is broken. Choose the red-line numbers first (error rate, latency, saturation), turn them into alert rules with thresholds you would actually act on, then push a deliberately bad build through the pipeline in a non-production subscription and watch the gate catch it. A rollback path you have never executed is a guess. Progressive delivery with no telemetry behind it is continuous deployment with extra waiting.

Try this

Pick one strategy for a lab service: rolling on AKS (Azure Kubernetes Service), a slot swap on App Service for the blue/green shape, or weighted traffic through Front Door or Traffic Manager. Then write down the exact rollback command, real resource group and app names filled in, while it is daytime and nothing is on fire. You want that line already written the night you need it at 2 a.m.

# Example: App Service slot swap is a blue/green-style flip
az webapp deployment slot swap -g rg-lab -n contoso-web --slot staging --target-slot production
# Rollback is the same command with slots reversed
az webapp deployment slot list -g rg-lab -n contoso-web -o table

$ az webapp deployment slot list -g rg-lab -n contoso-web -o table
Name         Status
-----------  --------
production   Running
staging      Running
 
# Sample output after swap — versions exchanged; no rebuild required.

Takeaway

Rolling, blue/green and canary each shrink the blast radius by paying a different bill: a mixed-version window, a duplicate environment, or telemetry you have to build and trust. Whichever you pick, it only helps if a metric decides whether the rollout continues and you have run the rollback at least once for practice.

Next step for your own pipeline: add a postRouteTraffic gate that queries Azure Monitor, so a canary throwing 5xx errors (the server-side failure codes) rolls itself back instead of waiting for somebody to notice a Slack message.

Related