Sync policies & self-heal

Auto-sync, prune, self-heal.

The last lesson was about a guard noticing the door is left open. This one is about what the guard does next. Argo CD runs a reconcile loop, a background check that constantly compares what Git says the cluster should look like against what is actually running, and flags any gap as OutOfSync. The sync policy is the standing order that decides the response: note it in the logbook and wait for a human, or walk over and lock the door itself. Leave the policy at its default and an OutOfSync application sits there, flagged, until someone runs argocd app sync by hand. Switch on automation and that same detection becomes continuous delivery: every commit to the tracked branch rolls out on its own, and the cluster is pulled back toward Git whenever it drifts. Three settings decide how far that self-correction goes.

The master switch

spec.syncPolicy.automated is the main breaker in the panel. Flip it, and Argo CD syncs on its own after any reconcile where Git and the cluster disagree, with no manual step. Inside it live two booleans that decide how aggressive that automatic sync gets. prune controls deletions. By default an automated sync only creates and updates, so a manifest (the YAML file that declares a resource) you delete from Git leaves its live object stranded in the cluster, still running but no longer declared anywhere. Set prune: true and a removal in Git becomes a deletion in the cluster. selfHeal controls the other direction. With it off, drift you introduce outside Git (an edited replica count, a hand-patched environment variable, a swapped image) is reported but left running. With it on, Argo CD rewrites the live object back to what Git declares on the next pass. automated on its own only tracks Git forward. Add prune and selfHeal and the cluster becomes a strict mirror of the repository: nothing lives that Git does not declare, and nothing differs from what Git says.

# turn on full automation, then confirm what stuck
argocd app set payments-api \
  --sync-policy automated --auto-prune --self-heal
 
argocd app get payments-api

Name:               argocd/payments-api
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          payments
URL:                https://argocd.example.com/applications/payments-api
Source:
- Repo:             https://github.com/acme/payments-config
  Target:           main
  Path:             overlays/prod
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (a1b2c3d)
Health Status:      Healthy
 
GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
       Service     payments   payments-api  Synced  Healthy        service/payments-api unchanged
apps   Deployment  payments   payments-api  Synced  Healthy        deployment.apps/payments-api unchanged

Self-heal is a remediation control

This is where sync policy stops being a deployment convenience and starts being a security control. Give an attacker a foothold with kubectl write access to a namespace and one of the first things they reach for is editing a live workload: swap the container image for a backdoored copy, add a sidecar (an extra container running alongside the main one) that reads your secrets, set privileged: true, or bump the replica count to mine cryptocurrency on your nodes. None of that touches Git. With selfHeal on, Argo CD watches the resources it manages, notices the live object no longer matches the declared one, and rewrites it back within seconds. The malicious edit is undone before it does much damage, and the sync is written to the application's history for you to find later.

# an attacker (or a panicked human) scales the deployment out of band
kubectl -n payments scale deployment/payments-api --replicas=10
kubectl -n payments get deploy payments-api -o jsonpath='{.spec.replicas}{"\n"}'
 
# a few seconds later, after Argo CD reconciles
kubectl -n payments get deploy payments-api -o jsonpath='{.spec.replicas}{"\n"}'

deployment.apps/payments-api scaled
10
3

You can watch self-heal fire in the deployment history. Every self-heal is a fresh sync, so several entries sitting at the same Git revision are a tell that something keeps mutating the live object. A tidy application shows one entry per real release. A run of same-revision syncs means either an attacker is poking at a workload or a controller you forgot about is fighting Argo CD for a field.

argocd app history payments-api

ID  DATE                           REVISION
0   2026-07-20 09:14:03 +0000 UTC  main (a1b2c3d)
1   2026-07-20 09:41:22 +0000 UTC  main (a1b2c3d)
2   2026-07-20 09:41:40 +0000 UTC  main (a1b2c3d)

Two limits keep self-heal honest as a defense. It only reverts fields on resources Argo CD already manages, so an attacker who creates a brand-new object that Git never declared (a rogue DaemonSet, meaning a workload that drops a pod on every node, or an extra ServiceAccount wired to cluster-admin) is not touched by self-heal at all. And prune only removes objects Argo CD itself created that later vanished from Git, so it will not clean up something an intruder made by hand either. Self-heal and prune keep declared resources honest; they do nothing about new, undeclared ones. For that you still want an admission controller like OPA (Open Policy Agent) Gatekeeper or Kyverno, policy engines that reject non-compliant resources at the Kubernetes API server (the control-plane front door every change to the cluster passes through) before they are ever created. Treat self-heal as the automatic reset button, not the lock on the door.

Prune and the mechanics of a sync

Beyond the three switches, syncOptions tune how each sync actually runs, and they apply to manual and automated syncs alike. They are the dials on the machine, not the timer that trips it: they change how a sync behaves, not when it happens. CreateNamespace=true has Argo CD create the destination namespace instead of failing when it is missing. ApplyOutOfSyncOnly=true skips resources that already match, so a large application reconciles faster. PruneLast=true holds deletions until everything else is healthy, so renaming a resource brings the replacement up before the old one comes down. PrunePropagationPolicy=foreground makes a deletion wait for its dependents (the pods under a Deployment, say) to go first instead of orphaning them. And retry decides what happens when a sync fails: limit caps the attempts, backoff grows the wait between them, so a transient failure like an image that has not been pushed yet, or a validating webhook (an external check the API server calls before admitting a change) that is briefly down, is retried with rising delay instead of giving up or hammering the API server.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/payments-config
    targetRevision: main
    path: overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true          # delete objects removed from Git
      selfHeal: true       # revert out-of-band drift back to Git
      allowEmpty: false    # never prune every resource at once (the default)
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
      - PruneLast=true
      - PrunePropagationPolicy=foreground
    retry:
      limit: 5
      backoff:
        duration: 5s       # wait before the first retry
        factor: 2          # double the wait each attempt
        maxDuration: 3m    # but never wait longer than this

A posture for each environment

The right policy is a spectrum you pick per environment, not one setting for everything. Development and preview clusters usually run fully automated with prune and selfHeal, so they stay hands-off and always fresh, and a mistake there is cheap because it corrects itself. Production often runs automated with prune but selfHeal off, or fully manual, so a person still confirms the rollout and can freeze it partway through an incident. The review gate then lives in branch protection on the Git repository, where a pull request has to be approved before it ever merges, which is exactly where GitOps (running your infrastructure from Git as the single source of truth) wants the gate to sit. Flipping a production application back to fully manual is one command.

# prod: auto-apply and prune, but let a human own live drift
argocd app set payments-api --sync-policy automated --auto-prune
argocd app get payments-api | grep 'Sync Policy'
 
# during an incident, freeze it completely (the default with no automated block)
argocd app set payments-api --sync-policy none
argocd app get payments-api | grep 'Sync Policy'

Sync Policy:        Automated (Prune)
Sync Policy:        <none>

When self-heal fights another controller

The fix is not to switch self-heal off. It is to stop Argo CD from managing that one field. Add a spec.ignoreDifferences entry that targets /spec/replicas, so Git owns the manifest and the HPA owns its one field. Here is the subtlety that trips people up: ignoreDifferences on its own only changes what Argo CD reports as OutOfSync in the diff. A real sync, including a self-heal sync, still applies the full manifest and would set replicas right back. To make a sync actually honor the ignore rule, add RespectIgnoreDifferences=true to syncOptions as well. The cleaner option, when you can take it, is to drop replicas from the manifest entirely so there is nothing to reconcile. Argo CD ships no default ignore for any of this; you configure it yourself.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: payments-api
      jsonPointers:
        - /spec/replicas            # let the HPA own the replica count
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true  # honor the ignore during sync, not only the diff
    automated:
      selfHeal: true

# after the change: let the HPA scale the deployment, then confirm Argo stays calm
kubectl -n payments get hpa payments-api
argocd app get payments-api | grep -E 'Sync Status|Health Status'

NAME          REFERENCE                TARGETS    MINPODS  MAXPODS  REPLICAS  AGE
payments-api  Deployment/payments-api  42%/70%    3        10       6         5d
Sync Status:        Synced to main (a1b2c3d)
Health Status:      Healthy

Before you switch selfHeal on for a production application, run argocd app diff on it once and read every field it wants to change. If that list includes anything a live controller writes (a replica count, an injected sidecar annotation, a mutated security context), settle the ownership with ignoreDifferences first. Self-heal is only as safe as your certainty about who owns each field.

Try this

Run argocd app get payments-api on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an empty render can delete your whole app. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related