Secrets & image updates

Secrets in GitOps; auto image bumps.

GitOps means running your whole cluster from files in a Git repository, with Git as the single source of truth: every Deployment, every ConfigMap, every setting that Argo CD reads and applies. That works right up until you hit the one thing you cannot write into a file other people can read: a password, an access token, a private key. Put a database password in a Git file and you have handed it to everyone who can clone the repo, and repos get cloned, forked, mirrored, and leaked. What you want is the shape of the secret in Git without the value.

The plaintext problem

A recipe card taped to the fridge is fine for the whole household to read. Your house key taped to that same card is not. Git is the fridge here, readable by everyone with access to the repo. The move every good approach makes is the same: tape a locked box to the fridge, put the secret inside the box, and keep the box's key somewhere else. What sits in Git becomes useless on its own, and something that holds the key, running inside your cluster, does the decrypting.

Three ways to keep the key out of Git

There are three common patterns. Sealed Secrets encrypts the value and keeps the ciphertext in Git. The External Secrets Operator keeps only a pointer in Git and fetches the value from a dedicated store. SOPS (Secrets OPerationS, a tool that encrypts the values inside a file in place) locks the values right there in the YAML file, where YAML is the text config format Kubernetes reads, and decrypts them at the moment Argo CD renders the manifest it applies to the cluster. They differ mainly in where the real secret lives and who holds the key that opens it.

Sealed Secrets in practice

Sealed Secrets, a controller from Bitnami (a program that runs inside your cluster and acts on the objects you create), works like a public mailbox slot on a locked box. Anyone can drop mail in: you encrypt your secret against a public certificate, and you can do that safely on your laptop or in CI (Continuous Integration, the automated build pipeline). Only the postmaster can open the box: a controller running in the cluster holds the matching private key and is the only thing that can decrypt. You commit the encrypted result, a SealedSecret, straight to Git.

kubectl create secret generic db-creds \
  --namespace payments \
  --from-literal=password='pv7!Kx2rL0' \
  --dry-run=client -o yaml \
| kubeseal --controller-namespace sealed-secrets --format yaml \
> sealedsecret.yaml

The --dry-run=client part builds the Secret object in memory and prints it without ever sending it to the cluster. kubeseal reads that, encrypts each value against the controller's public certificate, and writes a SealedSecret. Here is what lands in Git.

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-creds
  namespace: payments
spec:
  encryptedData:
    password: AgBy8h2mQf3kR9ZrLPx1t...oP4z9Qk2wD7fN==
  template:
    metadata:
      name: db-creds
      namespace: payments
    type: Opaque

Once Argo CD syncs this, the controller decrypts it into a real Secret. Check that both are present.

kubectl get sealedsecret,secret -n payments

NAME                                STATUS   SYNCED   AGE
sealedsecret.bitnami.com/db-creds            True     34s
 
NAME                     TYPE     DATA   AGE
secret/db-creds          Opaque   1      33s

Two things to hold onto. First, the SealedSecret is bound to its namespace and name by default (strict scope). Encrypt a value for payments/db-creds and it will refuse to decrypt as staging/db-creds. That stops someone copying your ciphertext into a namespace they control and reading it back. Second, notice the plain secret/db-creds in the output. Sealed Secrets protects the copy in Git, not the copy in the cluster. The decrypted value sits in etcd (the key-value database where Kubernetes keeps all of its state) exactly like any other Secret.

External Secrets: a reference, not the value

The External Secrets Operator (ESO, a controller that pulls secrets from an outside store) works like a coat-check ticket. The ticket in your pocket is worthless to a pickpocket. It means something only when you hand it to the attendant, who has the keys to the cloakroom. In Git you keep the ticket: an ExternalSecret object that names a secret and says where to find it. The real value lives in a dedicated store like HashiCorp Vault or AWS Secrets Manager, and ESO reads it and creates a normal Kubernetes Secret in the cluster.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: db-creds
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: payments/db
        property: password

secretStoreRef points at a ClusterSecretStore you defined once, which holds the address of Vault and the credentials ESO uses to log in. remoteRef is the coordinates of the value inside that store. refreshInterval: 1h tells ESO to re-read the store every hour. Ask the cluster how the sync is going.

kubectl get externalsecret db-creds -n payments

NAME       STORE   REFRESH INTERVAL   STATUS         READY
db-creds   vault   1h                 SecretSynced   True

That refresh line is the real difference from Sealed Secrets. Rotate the password inside Vault and, within the hour, ESO notices and updates the in-cluster Secret. No Git commit, no re-encrypt. With Sealed Secrets or SOPS the ciphertext is baked into Git, so rotating means encrypting again and committing again. Neither is better in the abstract: ESO trades a Git-only trail for live rotation and a hard dependency on the store staying reachable.

Automated image bumps

Now the second half of the lesson, next door but separate. Think of a standing order at the bakery: whenever a fresh loaf of your usual kind comes out of the oven, they set one aside for you, no phone call needed. Argo CD Image Updater is that standing order for container images. It watches your registry (the server that stores your built container images), and when a new tag appears that matches a policy you set, it updates the running image with no human editing YAML by hand. You steer it with annotations (small key-value notes attached to a Kubernetes object) on the Argo CD Application, the object that tells Argo CD what to deploy and from where.

metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=ghcr.io/acme/api:~1.4
    argocd-image-updater.argoproj.io/api.update-strategy: semver
    argocd-image-updater.argoproj.io/api.allow-tags: regexp:^\d+\.\d+\.\d+$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main

image-list names the image and a constraint: ~1.4 means the newest 1.4.x, so a jump to 2.0.0 needs a human. update-strategy: semver picks the highest tag that fits (semver is semantic versioning, the major.minor.patch numbering scheme). allow-tags with a regexp (a regular expression, a text-matching pattern) throws out anything that is not a clean release number, so a stray latest or 2.0.0-rc1 never qualifies. The write-back method matters most for security. git commits the change back to your repo as a small .argocd-source-api.yaml override file, so every automated bump shows up as a reviewable, revertible commit. The alternative, argocd, sets the override through the Argo CD API and leaves no Git trail, which quietly breaks the GitOps audit story.

kubectl -n argocd logs deploy/argocd-image-updater | tail -n 5

time="2026-07-20T09:14:02Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2026-07-20T09:14:03Z" level=info msg="Setting new image to ghcr.io/acme/api:1.4.2" alias=api application=api registry=ghcr.io
time="2026-07-20T09:14:04Z" level=info msg="Successfully updated image 'ghcr.io/acme/api:1.4.1' to 'ghcr.io/acme/api:1.4.2', now processing group write back"
time="2026-07-20T09:14:05Z" level=info msg="Committing 1 parameter change(s) for application api"
time="2026-07-20T09:14:06Z" level=info msg="Successfully pushed 1 change(s) to git branch main"

For a defender, the policy is where the safety lives. Without allow-tags, a mistaken push of a latest tag, or an attacker who gains write access to your registry and shoves a higher version number, can auto-deploy straight to production. Pin a tight tag pattern, keep the constraint narrow, and use Git write-back so a bad bump is one git revert away and shows up in review, not only in a running Pod (the running unit that wraps your container).

Progressive delivery with Argo Rollouts

An automated bump still deploys whatever passed your policy, and a build can pass every check and still be broken at runtime. Argo Rollouts is the taste test before you serve the whole pot. It replaces a plain Deployment with a Rollout that ships the new version to a small slice of traffic first (a canary, named for the bird miners once carried to sense bad air), watches real metrics, and widens the slice only if the numbers hold. Blue-green is the other mode: bring the new version up fully alongside the old one, then flip all traffic at once. The gate is an AnalysisTemplate that runs a query, usually against Prometheus (a metrics database), between canary steps.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="api",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{app="api"}[2m]))
      successCondition: result < 0.02

Each measurement compares the api service's share of 5xx responses (server errors, the 500-range HTTP status codes) against the two percent line in successCondition, watching the whole app while the canary takes its slice of traffic. failureLimit: 2 sets how much slack you give it, counted in failures: two readings can break the line and the rollout keeps going, but the third failed check fails the analysis. When that happens the rollout aborts on its own: traffic snaps back to the stable version and the new Pods scale down. You can watch a rollout in flight.

kubectl argo rollouts get rollout api -n payments

Name:            api
Namespace:       payments
Status:          Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          2/6
  SetWeight:     20
  ActualWeight:  20
Images:          ghcr.io/acme/api:1.4.1 (stable)
                 ghcr.io/acme/api:1.4.2 (canary)
Replicas:
  Desired:       5
  Current:       6
  Updated:       1
  Ready:         6
  Available:     6

Chain the two together and you get a pipeline that defends itself. Image Updater proposes a bump as a Git commit. Rollouts sends it to twenty percent of traffic, and the analysis watches error rates the way a smoke alarm watches for smoke. A broken or tampered build trips its own alarm and rolls back before it reaches the other eighty percent. Pin your tags tight, keep the write-back in Git, and put an analysis gate in front of every automated bump, so the worst a bad image can do is fail on a small slice of traffic and vanish.

Try this

Run kubectl get sealedsecret,secret -n payments on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: where the trust really lives. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related