BlogCI/CD

Blue-green and canary deploys from your pipeline

Ship with zero downtime and instant rollback by shifting traffic between versions instead of replacing in place.

Oct 22, 2024·4 min readAdvanced·By the SecOpsLog team · command-tested

Rolling updates replace pods in place — if the new version is bad, part of your fleet is already serving errors before you notice. Blue-green keeps two full versions alive and flips traffic between them; canary sends a slice of traffic to the candidate while metrics decide promotion. Rollback becomes a routing change measured in seconds, not a redeploy praying the old artifact still exists.

You will deploy green beside blue, smoke-test without live traffic, shift the router or Service selector, and automate canary steps with analysis hooks. Budget briefly for double capacity during cutover — the cost of an extra replica set is cheaper than the revenue lost during a bad rolling deploy. The pattern works on Kubernetes Services, load balancers, and GitLab environments — pipeline orchestration details align with Secure CI/CD with GitLab and progressive delivery modules there.

Blue-green cutover and rollback

Keep blue idle until green is proven. Rollback is a selector flip.

Blue (live) current production pods labeled env=blue Service / Ingress selector flip = cutover Green (idle) new version, no traffic smoke then switch 1 Cutover point selector at green 100% traffic in one step 2 Watch errors · latency · saturation soak window before teardown 3 Rollback flip selector back to blue instant — no rebuild Keep blue idle until green proves stable. Rollback is routing, not redeploy. blue ←→ Service selector ←→ green
Blue — liveGreen — candidateRollback — flip back
Blue-green cutover with instant rollback

Keep blue idle after cutover until green proves stable — rollback is pointing traffic back, not rebuilding.

1Blue livecurrent production2Deploy greensame capacity, no traffic3Smoke testssynthetic checks on green4Switch router100% to green5Watch metricserrors, latency, saturation6Blue idlerollback target7Decommission blueafter soak window

The switch is routing, not redeploy

At its simplest on Kubernetes, blue-green is two Deployments or ReplicaSets behind one Service — flip the selector label and endpoints update. On AWS or GCP the same idea is two target groups behind an ALB or load balancer rule. The pipeline job that matters is the traffic shift, not another kubectl apply under fire. Document the rollback command in the runbook and in the pipeline itself so on-call does not invent flags at 2am.

deploy-blue-green.sh
# green already deployed and smoke-tested
kubectl patch service api -n prod -p \
'{"spec":{"selector":{"version":"green"}}}'
# rollback: patch selector back to version=blue

Canary when all-at-once is too bold

Argo Rollouts (or Flagger) automates weighted traffic shifts and metric analysis — promote 10%, pause, check Prometheus error rate, then 50%, then 100%. A failed analysis at 10% only ever touched 10% of users. Encode steps in the Rollout spec so promotion is policy, not a human remembering to watch Grafana. Wire analysis templates to symptoms users feel — 5xx rate, p99 latency — not cause metrics like CPU that lie during partial outages.

rollout.yaml
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 5m }
- analysis:
templates:
- templateName: error-rate
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
bash — confirm traffic splitlive
kubectl argo rollouts get rollout api -n prod
Step: setWeight 10% Phase: Paused
kubectl get endpoints api -n prod -o wide
stable and canary pods both receive weighted traffic
abort rollout — instant revert to stable ReplicaSet
Three deploy strategies
Rolling (default)
Replace pods in place
No extra capacity
Rollback = redeploy
Simplest, highest blast radius
Blue-green / canary
Two versions live briefly
Shift traffic, not rebuild
Instant rollback path
Needs metrics + 2x capacity
Schema migrations break naive blue-green
If green expects a database column blue does not have, cutting traffic over causes simultaneous outages. Make migrations backward compatible (expand-contract) or run canary only after schema is live on both code paths. Progressive delivery does not forgive incompatible DDL.

Where this goes next

Traffic shifting only helps when health checks and graceful shutdown drain connections — otherwise the new version drops in-flight requests during the cutover. Wire deploy pipelines to require green CI scans, signed images, and post-promotion smoke tests before the router moves. Pair progressive delivery with review apps so stakeholders validate features in isolation before any percentage of production traffic sees them. Secure CI/CD with GitLab ties environments, progressive delivery, and supply-chain gates into one pipeline story.

Go deeper in a courseSecure CI/CD with GitLabEnvironments, deploy strategies, protected pipelines, and scanning gates.View course

Related posts