Canary, blue/green & rollback

Ship safely, revert instantly.

It is Friday afternoon. Your pipeline builds image sha256:9f3c1b (a digest, the content hash that names one exact image and nothing else), runs it past SAST (static application security testing, which reads your source for bug patterns), secret detection and container scanning, signs it with cosign, and merges a one-line digest bump into the deploy repo. Every gate is green. Argo CD, the agent that keeps the cluster matching what is in Git, syncs the change. Forty seconds after the new version is carrying 100% of traffic, checkout p99 latency (the wait suffered by the slowest one request in every hundred) triples and the 5xx rate (the share of requests the server fails outright) climbs to 4%. No test caught it, because nothing is wrong with the code on its own. It only buckles when thousands of real users arrive at once. This is the last lesson in the delivery arc. You have built, scanned, signed, attested and gated the artifact, and progressive delivery is how you put it in front of users without wagering the whole service on one move. A kitchen trying a new dish does not send it to every table at once. A few plates go out, someone watches the faces, then the call gets made. Same idea here: give the new version a thin slice of live traffic, measure it against real signals, and let a bad release convict itself while it can still only annoy a handful of people.

Canary vs blue/green

Coal miners used to carry a canary down the shaft. If the bird went quiet, the air was bad, and everyone climbed out before a person got hurt. A canary release borrows the name and the logic. You route a small slice of live traffic, say 5%, to the new version (the canary) while the old version (the stable) keeps serving everyone else. You watch the canary's metrics, ramp 5 → 25 → 50 → 100% only while they hold, and abort the moment they sag. Blue/green works differently. You bring the whole new version up beside the old one, throw a single switch, and move 100% of traffic across, keeping the old copy running and warm so going back is one more flip. Canary buys safety with time: trouble shows up while few users are exposed, but a full rollout can take an hour. Blue/green is instant in both directions and all-or-nothing, it needs double the capacity while both copies are up, and it gives you no gradual signal to judge by. One thing makes either strategy trustworthy: a metric gate. That is an automated check that promotes on good numbers and rolls back on bad ones, so the release decision is measured rather than guessed.

That third branch is the one teams forget. A query that comes back empty tells you nothing at all. Maybe the exporter publishing the metrics fell over. Maybe the canary is serving so few requests that there is nothing worth averaging. Either way the honest answer is 'unknown', and a sane gate reads unknown as 'do not promote'. It holds at the current weight and waits for a person, rather than gambling in either direction.

The canary as code: Argo Rollouts

Argo Rollouts is a Kubernetes controller, a small program that sits in the cluster watching objects and pushing reality toward what they describe. It replaces the built-in Deployment with a Rollout object that understands canary and blue/green natively. Its canary strategy reads like a flight plan: an ordered list of steps, setWeight and pause, with analysis stops written in between. An analysis step starts an AnalysisRun, which asks a metrics provider (Prometheus, Datadog, CloudWatch) a question and checks the answer against a successCondition. Good answer, the rollout moves on. Bad answer, it aborts and pulls every request back to stable without waiting for you. The traffic split itself happens down where user requests actually flow: Rollouts programs an ingress controller (NGINX, or an AWS ALB, Application Load Balancer) or a service mesh (Istio, or Linkerd through SMI, the Service Mesh Interface). canaryService and stableService are the two Services pointing at the two ReplicaSets it shifts weight between. The gate lives in the manifest, in Git, next to the workload it guards, so nobody has to remember a runbook step.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: payments, namespace: payments }
spec:
  replicas: 10
  selector: { matchLabels: { app: payments } }
  template:
    metadata: { labels: { app: payments } }
    spec:
      containers:
        - name: payments
          image: registry.acme.internal/payments@sha256:9f3c1b   # the digest CI signed
  strategy:
    canary:
      canaryService: payments-canary
      stableService: payments-stable
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - setWeight: 25
        - analysis:                        # metric gate on the 25% canary
            templates: [{ templateName: error-rate }]
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: error-rate, namespace: payments }
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] < 0.01   # < 1% 5xx, or the run fails
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="payments",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{app="payments"}[2m]))

Apply that once and the machinery runs itself. Every new signed digest that lands in the manifest kicks off a canary. To watch one move, use the rollouts kubectl plugin. It prints the current step, the live traffic weight, both ReplicaSets and the state of each AnalysisRun, which is how you tell a canary that is progressing from one that has quietly stalled.

$ kubectl argo rollouts get rollout payments -n payments
Name:            payments
Namespace:       payments
Status:          ◌ Progressing
Strategy:        Canary
  Step:          4/7
  SetWeight:     25
  ActualWeight:  25
Images:          payments@sha256:9f3c1b (canary)
                 payments@sha256:1a7be4 (stable)
 
NAME                             KIND         STATUS        AGE  INFO
⟳ payments                       Rollout      ◌ Progressing 8m
├──# revision:6
│  ├──⧉ payments-7d9c (canary)   ReplicaSet   ✔ Healthy     4m   canary
│  └──α payments-7d9c-4-analysis AnalysisRun  ◌ Running     2m   ✔ 3
└──# revision:5
   └──⧉ payments-5f8b (stable)   ReplicaSet   ✔ Healthy     3d   stable

Flagger comes at the same problem from the other side. It leaves your Deployment untouched and drives the canary from a separate Canary resource, but the metric-gate idea underneath is identical. Pick Rollouts when you want the delivery strategy versioned alongside the workload. Pick Flagger when you would rather leave your Deployments alone. Rollouts does blue/green too (activeService and previewService, with a promotion step) if you want the instant all-or-nothing switch, and the analysis hooks work the same way there. Whichever you choose, size the pause durations to your real traffic: long enough to collect a sample worth trusting, short enough that the full rollout finishes while anyone still cares.

Where GitLab CI fits

CI never drives the canary. That is the controller's job. What CI (continuous integration, the pipeline that builds and checks every commit) does is produce the input, and hold the gate shut until a person opens it. In a GitOps setup the pipeline's last step bumps the signed digest in the deploy repo, Argo CD syncs it, and the Rollout starts. In a push setup, a deploy job runs the promotion command directly. Either way, give the job a hotel key card that stops working at checkout, not a brass key cut for the front door. id_tokens mints a short-lived JWT (JSON Web Token, a signed blob that says 'this job, in this project, on this branch'), and Vault trades it for a scoped token with a clock running on it, using OIDC (OpenID Connect, the standard for one system vouching for another's identity). Then scope the job to protected branches only. A merge request from a fork runs with the fork's permissions and receives none of your protected variables, so it can neither reach the token's audience nor write to the deploy repo. That is deliberate. A stranger's pipeline must never be able to promote a canary into production.

promote-canary:
  stage: deploy
  image: registry.acme.internal/ci/argo-rollouts:latest
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.acme.internal
  environment:
    name: production
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual          # a human opens the gate; the analysis decides the rest
  script:
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=deploy jwt=$VAULT_ID_TOKEN)"
    - vault kv get -field=kubeconfig secret/prod/kube > "$KUBECONFIG"
    - kubectl argo rollouts promote payments -n payments

$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=deploy jwt=$VAULT_ID_TOKEN)"
$ vault kv get -field=kubeconfig secret/prod/kube > "$KUBECONFIG"
$ kubectl argo rollouts promote payments -n payments
rollout 'payments' promoted
Cleaning up project directory and file based variables
Job succeeded

A manual button can ask a question, or it can check an answer. Make it check. Before that job flips the next weight, have it verify the running canary's image signature and its SLSA provenance (Supply-chain Levels for Software Artifacts, a signed record of which pipeline built this image from which commit) against your own CI identity. Then the person clicking 'promote' is asserting something real: this canary is measurably healthy, and it is the same artifact we built and signed. The metric gate proves the release behaves. The signature gate proves it is ours. Gate on evidence, never on someone remembering to look.

When the analysis fails

Here is the payoff. The moment the AnalysisRun breaches its successCondition, Rollouts aborts. Canary weight drops to zero and stable is carrying every request again within seconds, an automatic rollback decided by the measured signal with nobody woken up. But an abort moves traffic, not intent. A tripped breaker cuts the power while the faulty heater stays plugged into the wall. The Rollout spec, and the Git commit behind it, still name the bad digest, so the next sync or a retry will happily try it again. To make the rollback stick, fix the source of truth: kubectl argo rollouts undo rolls the revision back, or in GitOps you revert the commit that bumped the digest and let Argo CD reconcile to the last known-good image. Traffic back on healthy code in under a minute is the whole reason the gate exists. The abort buys that minute, and reverting the desired state is what keeps it.

$ kubectl argo rollouts get rollout payments -n payments
Name:            payments
Status:          ✖ Degraded
Message:         RolloutAborted: metric "error-rate" assessed Failed
                 due to failed (2) > failureLimit (1)
Strategy:        Canary
  Step:          4/7
  ActualWeight:  0                     # traffic shifted back to stable automatically
 
NAME                             KIND         STATUS       AGE  INFO
⟳ payments                       Rollout      ✖ Degraded   11m
├──# revision:6
│  ├──⧉ payments-7d9c (canary)   ReplicaSet   • ScaledDown 7m   canary
│  └──α payments-7d9c-4-analysis AnalysisRun  ✖ Failed     3m   ✖ 2
└──# revision:5
   └──⧉ payments-5f8b (stable)   ReplicaSet   ✔ Healthy    3d   stable
 
$ kubectl argo rollouts undo payments -n payments   # revert desired state to last good
rollout 'payments' undone

Try this

Run kubectl argo rollouts get rollout payments -n payments on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a 5% canary on a quiet service measures noise, not health. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related