Progressive delivery

Blue/green, canary, rolling, automated rollback.

Advanced30 min · lesson 13 of 15

A road crew opening a fresh lane on a motorway has two choices. Wave every car onto the new asphalt at once and hope the surface holds. Or cone off one lane, let a trickle across, watch for potholes, and only then move everyone over. Progressive delivery is that second habit turned into machinery. You hand a new version of your software to a growing slice of real traffic while automated checks decide, minute by minute, whether to keep ramping or back out.

One number decides what a bad release costs you: blast radius, the share of users a broken version can reach before something stops it. Shipping often is only safe when every release keeps that number small and can be undone with nobody awake to press the button. Everything below wires four rollout strategies and alarm-triggered rollback together in AWS CodeDeploy (the Amazon Web Services tool that pushes new versions out and controls how traffic finds them), so you can deploy several times a day without booking an outage for each one.

Four strategies on the same dial

All-at-once swaps every instance in a single step. It is the quickest option and the worst one under pressure, because a defect reaches 100% of users the moment it lands and the only way back is another full redeploy. Rolling replaces instances in batches, say two at a time, while the rest keep serving. Blast radius drops to one batch. The catch is that for a while you are running two versions of your code side by side against the same database and the same API (application programming interface) contracts, so old and new have to tolerate each other in both directions.

Blue/green builds a complete second environment (green) next to the one currently serving (blue), moves traffic across once green looks healthy, and leaves blue running so you can flip back instantly. No real user touches green before the cutover, so blast radius until that moment is zero, and rollback takes seconds. You pay for that by running two full fleets during the window. Canary sends a small slice of live traffic, usually 10%, to the new version first, watches the metrics through a bake window, and ramps the rest of the way only if the signals stay clean. Canary and blue/green are what production deserves. All-at-once belongs in dev.

How CodeDeploy moves the traffic

CodeDeploy stores each strategy as a deployment config, a named schedule for how traffic moves. AWS ships predefined ones for its three compute platforms: ECS (Elastic Container Service, for containers), Lambda (functions that run without servers you manage), and EC2 (Elastic Compute Cloud, plain virtual machines). Underneath there are only two routing *types*. TimeBasedCanary shifts one percentage, holds it for an interval, then sends the rest. TimeBasedLinear adds the same fixed percentage every interval. When none of the built-ins matches how nervous a service makes you, write your own with a smaller first step.

list-and-create-deployment-config.sh
$ aws deploy list-deployment-configs --query 'deploymentConfigsList' --output json
[
"CodeDeployDefault.OneAtATime",
"CodeDeployDefault.HalfAtATime",
"CodeDeployDefault.AllAtOnce",
"CodeDeployDefault.ECSAllAtOnce",
"CodeDeployDefault.ECSLinear10PercentEvery1Minutes",
"CodeDeployDefault.ECSCanary10Percent5Minutes",
"CodeDeployDefault.ECSCanary10Percent15Minutes",
"CodeDeployDefault.LambdaCanary10Percent5Minutes",
"CodeDeployDefault.LambdaLinear10PercentEvery1Minute"
]
# Want a gentler first step than the built-in 10%? Define a 5%/5-min canary:
$ aws deploy create-deployment-config \
--deployment-config-name Canary5Percent5Minutes \
--compute-platform ECS \
--traffic-routing-config \
'{"type":"TimeBasedCanary","timeBasedCanary":{"canaryPercentage":5,"canaryInterval":5}}'
{
"deploymentConfigId": "acd3e1a0-8b1f-4d2c-9a77-2f5b0e4c1d9a"
}

For an ECS blue/green deployment the plumbing is worth knowing in detail. CodeDeploy starts a green task set (a second running copy of your containers) behind a second target group, which is the pool of endpoints a load balancer is allowed to send requests to. The production listener on your ALB (Application Load Balancer, the front door that spreads incoming requests across healthy targets) keeps pointing at the blue target group. Optionally a test listener points at green on a private port, so you can hit the new version yourself while nobody else can. A BeforeAllowTraffic lifecycle hook runs your smoke tests against green *before* a single user reaches it. Only once that passes does CodeDeploy begin moving the production listener's forward rule from the blue target group to green, on the schedule your deployment config defines. The stretch where both versions are live and the alarms are being watched is the bake window. That wait is the safety you are buying.

Wire rollback to signals users can feel

Choosing a strategy is half the safety story. The other half is automated rollback bolted to what you actually measure. Two settings on the deployment group do it. An auto-rollback-configuration lists the *events* that trigger a revert (DEPLOYMENT_FAILURE and DEPLOYMENT_STOP_ON_ALARM). An alarm-configuration names the CloudWatch alarms (CloudWatch is the AWS metrics and alarms service) to watch while the rollout runs. The moment any listed alarm enters ALARM, CodeDeploy stops shifting traffic and reverses. Point those alarms at what users feel: the rate of 5xx responses (server-side errors) and p99 latency (how slow the worst 1% of requests are). CPU and memory lag the actual harm, so they make poor triggers.

wire-rollback.sh
# The rollback trigger: 5xx on the ALB target group, one breaching minute is enough
$ aws cloudwatch put-metric-alarm \
--alarm-name web-api-5xx \
--namespace AWS/ApplicationELB \
--metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=app/web-api-alb/50dc6c495c0c9188 \
--statistic Sum --period 60 --evaluation-periods 1 --threshold 5 \
--comparison-operator GreaterThanThreshold --treat-missing-data notBreaching
# Attach the alarm + auto-rollback to the group your CDK/CloudFormation stack created:
$ aws deploy update-deployment-group \
--application-name web-api \
--current-deployment-group-name web-api-ecs \
--auto-rollback-configuration 'enabled=true,events=DEPLOYMENT_FAILURE,DEPLOYMENT_STOP_ON_ALARM' \
--alarm-configuration 'enabled=true,alarms=[{name=web-api-5xx},{name=web-api-p99}]'
# (no output on success -> exit code 0)

Now the payoff. You start a deployment, green begins returning errors, the alarm crosses its threshold while the bake window is still open, and CodeDeploy turns the release around. For blue/green that turn is one flip of the listener back to the blue target group, which never stopped running, so traffic is healthy again in seconds even though CodeDeploy records the reversal as a *new* deployment with its own ID. get-deployment tells you exactly what happened and why:

deploy-and-observe-rollback.sh
$ aws deploy create-deployment \
--application-name web-api --deployment-group-name web-api-ecs \
--revision file://appspec-revision.json \
--query 'deploymentId' --output text
d-A1B2C3D4E
# 10% shifts to green; the 5xx alarm fires during the bake window:
$ aws deploy get-deployment --deployment-id d-A1B2C3D4E \
--query 'deploymentInfo.{status:status,config:deploymentConfigName,rollback:rollbackInfo}'
{
"status": "Stopped",
"config": "CodeDeployDefault.ECSCanary10Percent5Minutes",
"rollback": {
"rollbackTriggeringDeploymentId": "d-A1B2C3D4E",
"rollbackDeploymentId": "d-F5G6H7I8J",
"rollbackMessage": "Deployment was stopped and rolled back because alarm web-api-5xx entered ALARM state."
}
}
ECS blue/green traffic-shifting architecture
Application Load Balancer
Production listener :443
Live users; forward rule moves blue -> green on the config schedule
Test listener :8443
Validation traffic to green before any production shift
Blue (current v1)
Target group: blue
100% -> 0% of production traffic
Task set v1
Kept warm the whole bake window for instant flip-back
Green (new v2)
Target group: green
0% -> 100% as the canary/linear steps advance
BeforeAllowTraffic hook
Lambda smoke test must pass before user traffic starts
Rollback guard
CloudWatch alarms: 5xx, p99
Watched only while the deployment is in progress
DEPLOYMENT_STOP_ON_ALARM
Flips the listener back to blue in seconds
Blue stays live and warm until green owns 100%, so rollback is a listener flip, not a cold restart.

Serverless functions get the same machinery without the command-line wiring. In SAM (Serverless Application Model, a shorthand template language that expands into CloudFormation) or in the CDK (Cloud Development Kit, which generates those templates from real code), AutoPublishAlias plus a DeploymentPreference block is the whole ask. SAM then creates the CodeDeploy application, the canary schedule, the alarm binding and the hooks, all pointed at a Lambda alias. A PreTraffic hook checks the new version before the alias starts weighting any traffic toward it.

template.yaml
Resources:
CheckoutFn:
Type: AWS::Serverless::Function
Properties:
AutoPublishAlias: live
DeploymentPreference:
Type: Canary10Percent5Minutes # 10% for 5 min, then 100%
Alarms:
- !Ref FnErrorsAlarm # auto-rollback if this trips
Hooks:
PreTraffic: !Ref SmokeTestFn # validate before shifting the alias
# $ sam deploy --no-confirm-changeset
# CloudFormation outputs: CheckoutFn.Alias live -> version 7, DeploymentPreference active

What it costs, and where it breaks

Blue/green runs two full fleets for a while, so budget roughly double the compute during every cutover and check you have headroom under the limits that actually bite. The Fargate On-Demand vCPU resource count is the first (Fargate is the mode where AWS runs your containers without you owning any servers, and a vCPU is one virtual processor). Its documented default can be as low as 6 vCPU per Region on a brand-new account. It is adjustable, and Fargate raises it on its own as your steady-state usage grows. After that come your ECS service's task ceiling and the capacity of the ALB target group. If the account is sitting near any of those, the green task set never launches and the deployment errors out before a single request moves, which is a wretched thing to learn during an incident. Raise the quotas *before* your first blue/green. CodeDeploy's own ceiling is generous, up to 1,300 concurrent deployments per account (an adjustable soft limit), with deployment configs and groups sitting comfortably inside normal account limits. What stops you is almost always compute, not CodeDeploy.

The real dial here is detection quality against rollout speed. A longer bake window and a smaller first slice give your alarms more time to spot a regression while fewer people are exposed. They also stretch every deploy from minutes into tens of minutes, which slows the team down and keeps two versions of your code talking to the same database for longer. Set the interval by how fast your worst failure actually shows itself. If a bad build spikes 5xx inside a minute, a 5-minute canary is plenty. If the damage only appears under sustained load, you need a longer bake, or synthetic traffic pushed through green while it bakes.

An alarm slower than the bake window never gets to fire
DEPLOYMENT_STOP_ON_ALARM can only pull a release back while traffic is still shifting. An alarm confirms a breach after period multiplied by evaluation-periods of bad data. If that span is longer than your canary or linear interval, the rollout reaches 100% and is marked Succeeded before the alarm has made up its mind, and a broken release ships with nothing standing in its way. Keep the alarm's evaluation window shorter than the bake time (period=60 with evaluation-periods=1 against a 5-minute canary works), and set treat-missing-data to notBreaching so a metric that stops reporting altogether cannot quietly hide the failure.

Canary and linear rollouts only earn their keep if the metrics track what users feel. Shifting 10% of traffic to a build that never emits the metric your alarm reads is theatre with a progress bar.

All-at-once is fine for a batch worker nobody is waiting on. It is reckless for an API that customers are hitting right now. Pick the strategy from the blast radius you can afford, not from how confident a developer feels at five o'clock on a Friday.

Try this

Look at the shape of a built-in canary config, then at the record of a real blue/green or canary run that used one, on a lab app you control.

terminal
aws deploy get-deployment-config --deployment-config-name CodeDeployDefault.ECSCanary10Percent5Minutes \
--query 'deploymentConfigInfo.{Name:deploymentConfigName,Traffic:trafficRoutingConfig}' --output json
aws deploy get-deployment --deployment-id d-CANARY1 \
--query 'deploymentInfo.{Status:status,Config:deploymentConfigName,Start:createTime}' --output table
output
{
"Name": "CodeDeployDefault.ECSCanary10Percent5Minutes",
"Traffic": {"type": "TimeBasedCanary", "timeBasedCanary": {"canaryPercentage": 10, "canaryInterval": 5}}
}
Succeeded | CodeDeployDefault.ECSCanary10Percent5Minutes | 2026-07-23T17:00:00Z

Takeaway

Remember: progressive delivery shrinks blast radius only when the traffic shifts are gated on alarms that measure something real. Gate them on nothing and all you have automated is hope.

Next: take your slowest production path, decide canary or linear for it, and write down which alarm is allowed to abort the shift.

Quick check
01An ECS canary (ECSCanary10Percent5Minutes) holds 10% of traffic for 5 minutes, then shifts to 100%. Its rollback alarm is set to period=300s with evaluation-periods=3. A release that fails every single request still reaches 100% and never rolls back. Why?
Correct — DEPLOYMENT_STOP_ON_ALARM is armed only while the deployment is in progress. If the alarm takes longer to confirm a breach than the bake window lasts, rollback never gets its chance. Shorten the alarm to period=60 with evaluation-periods=1, or lengthen the canary interval.
Incorrect — No. DEPLOYMENT_STOP_ON_ALARM is precisely the event that ties CloudWatch alarms to rollback. DEPLOYMENT_FAILURE is a separate trigger you list alongside it.
Incorrect — No. Alarm-based rollback works with either routing type, canary or linear. The shape of the traffic schedule does not switch off the alarm-configuration.
Incorrect — No. That metric is published every minute in real time while the deployment runs. The data is there. The alarm cannot gather enough datapoints fast enough to confirm the breach.
02CodeDeploy offers exactly two traffic-routing types, TimeBasedCanary and TimeBasedLinear. What makes a config TimeBasedLinear?
Incorrect — That describes TimeBasedCanary, which holds a single percentage before releasing the remainder.
Incorrect — That is an all-at-once deployment, not a linear traffic-routing config.
Incorrect — Both routing types run off a clock. Neither one inspects request content.
Correct — Linear climbs in equal steps, while Canary holds one percentage and then releases the rest.
03A regulated payments service has three requirements: no user may touch the new version before cutover, the team must be able to go back to the previous version within seconds if it misbehaves, and they have agreed to pay for a second full fleet while the deployment runs. Which strategy fits best?
Incorrect — It hands a defect to every user at once, and getting back means another full redeploy rather than a flip.
Incorrect — A batch of live users sees the new version and you run mixed versions for a while, so exposure before cutover is not zero.
Correct — Nobody reaches green before the cutover, and rollback is a listener flip back to the blue fleet that never stopped running. The price is two fleets for the duration, which this team has accepted.
Incorrect — A canary puts roughly 10% of live users on the new version first, so exposure before full rollout is not zero.

A release you can reverse in seconds still tells you nothing about a data centre catching fire. The green task set you shifted traffic to lives in Availability Zones that can go dark and in a Region that can have a very bad day. Next, HA & DR automation (high availability and disaster recovery) takes the same pieces you wired here, health-gated traffic shifting, warm standbys and alarm-driven automation, and aims them at failing infrastructure rather than failing code.

Related