The reconcile loop & drift

How Argo detects and corrects.

A home thermostat runs the same small loop forever. It reads a target (the temperature you set on the dial), reads the current value (what the room actually is), compares the two, and nudges the heat to close the gap. Argo CD (the CD stands for continuous delivery, the practice of rolling your declared changes out to running systems automatically) works the same way, only the room is a Kubernetes cluster (the fleet of machines that runs your containers). Its dial is your Git repository (the version-control system that stores your configuration as files). The manifests sitting in Git are the desired state. Its thermometer is the live cluster. The reconcile loop is that endless measure-then-compare cycle, and drift is the gap it finds the moment the cluster stops matching Git.

The loop that never sleeps

The work is done by a component called the application-controller (the brain that manages every Application object you register with Argo). For each Application it runs a reconciliation. It asks the repo-server (a helper whose only job is to pull from Git and render templates, either Helm charts or Kustomize overlays, the two common ways to generate Kubernetes YAML) for the desired manifests (the YAML files, a plain indentation-based text format, that declare what should exist) at the target revision. It turns those into the Kubernetes objects Git says should exist, pulls the matching live objects out of the cluster, and compares the two sets field by field. That comparison is the whole game.

How often does it run? A setting named timeout.reconciliation in the argocd-cm ConfigMap (a Kubernetes object that holds plain configuration as key-value pairs) controls it, and it defaults to 180 seconds. That number is a backstop, not the full story. The controller also keeps a live cache of cluster objects using Kubernetes watches (a streaming subscription that pushes every change the instant it happens, instead of asking again and again), so when someone edits a managed object by hand, Argo usually notices within a second or two and re-runs the comparison. It also honors Git webhooks (a webhook is an automated message one system fires at another the moment an event happens, here a push landing in your repo), so a commit can trigger a reconcile right away. The 180-second timer is the guarantee of a full re-read even when a watch event got dropped or Git changed with no webhook wired up.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # max time allowed between reconciliations; a safety backstop,
  # not how fast drift is normally caught (watches do that in ~seconds)
  timeout.reconciliation: 180s

Seeing the verdict

The command you will live in is argocd app get. It prints the current desired-versus-live comparison for one Application, with a per-resource table underneath.

argocd app get guestbook

Name:               argocd/guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          guestbook
URL:                https://argocd.example.com/applications/guestbook
Source:
- Repo:             https://github.com/example/guestbook.git
  Target:           main
  Path:             manifests
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from main (a1b2c3d)
Health Status:      Healthy
 
GROUP  KIND        NAMESPACE  NAME       STATUS     HEALTH   HOOK  MESSAGE
       Service     guestbook  guestbook  Synced     Healthy        service/guestbook unchanged
apps   Deployment  guestbook  guestbook  OutOfSync  Healthy        deployment.apps/guestbook configured

Read the top block first. Sync Status: OutOfSync means at least one managed field in the cluster no longer matches Git. Health Status is a separate axis (is the app actually working right now), and it can read Healthy while the app is OutOfSync, exactly as it does here. The table names the culprit: the Deployment differs, the Service is fine. How does Argo know which objects are its to compare? It stamps a tracking marker on everything it applies (by default the label app.kubernetes.io/instance, or an annotation instead if you switch resourceTrackingMethod). Anything without that marker is invisible to the loop. That cuts two ways for defense: an attacker who creates a brand-new rogue Deployment produces no drift at all, because Argo was never told to own it (unless you turn on orphaned-resource monitoring, which is off by default).

Reading the diff

OutOfSync tells you that something differs. argocd app diff tells you exactly what. It renders the desired object from Git, normalizes both sides (dropping fields that carry no real meaning, like server-set defaults), and prints the difference line by line. It also exits non-zero when drift exists, which makes it a clean gate in a continuous-integration pipeline (the automated build-and-test run that fires on every commit): argocd app diff guestbook && echo in-sync || echo drift-detected.

# what differs between Git (desired) and the cluster (live)?
argocd app diff guestbook
echo "exit code: $?"

===== apps/Deployment guestbook/guestbook ======
@@ -30,4 +30,4 @@
   spec:
     progressDeadlineSeconds: 600
-    replicas: 5
+    replicas: 3
     revisionHistoryLimit: 10
exit code: 1

The minus line is the cluster as it stands right now (5 replicas). The plus line is what Git says it should be (3). Someone scaled the Deployment up by hand. That one screen is the security payoff of the whole loop. An attacker who lands in your cluster and tampers with a workload (swaps the container image for a backdoored build, or flips a securityContext field Git had locked down, like runAsNonRoot from true to false, where securityContext is a container's block of security settings) leaves this exact footprint. Wire an alert to OutOfSync and the reconcile loop turns into a tripwire that fires on any change you did not commit to Git.

Detection is not correction

Here is where people trip. The loop finds drift. It does not fix it. Left alone, that Deployment keeps running 5 replicas and Argo keeps reporting OutOfSync, patiently, forever. Closing the gap is a separate, deliberate action called a sync. argocd app sync re-applies the desired manifests and overwrites whatever changed. Making Argo do that on its own, with nobody typing sync, is self-heal, which the next lesson covers.

# re-run the comparison NOW instead of waiting for the 180s timer
argocd app get guestbook --refresh
 
# also drop the rendered-manifest cache (re-render Helm/Kustomize from scratch)
argocd app get guestbook --hard-refresh

Name:               argocd/guestbook
Project:            default
Sync Status:        OutOfSync from main (a1b2c3d)
Health Status:      Healthy
 
GROUP  KIND        NAMESPACE  NAME       STATUS     HEALTH   HOOK  MESSAGE
       Service     guestbook  guestbook  Synced     Healthy        service/guestbook unchanged
apps   Deployment  guestbook  guestbook  OutOfSync  Healthy        deployment.apps/guestbook configured

Notice what the refresh did. It recomputed the comparison and still reports OutOfSync. The 5 replicas are untouched. Refresh answered the question 'is there drift?', which is a different question from 'fix the drift.'

Teaching Argo to ignore benign drift

Not every difference is drift you want to chase. The Kubernetes API server fills in defaults for fields you never set. Some controllers legitimately own a field that also appears in your manifest. The classic case is the Horizontal Pod Autoscaler (HPA, the component that raises and lowers replica counts based on load), which owns spec.replicas: Git says 3, the HPA says 5 under traffic, and both are right. Argo reads that as a difference and will flag the app OutOfSync forever, because every sync re-applies your Git value of 3 and the HPA instantly sets it back to 5. You get a tug-of-war that never ends.

The escape hatch is spec.ignoreDifferences on the Application. You name the paths Argo should leave out of the comparison, three ways: by JSON pointer (a slash-delimited path into the object, like /spec/replicas), by a jq path expression (a query written in jq, a small language for picking values out of JSON, handy for a key buried inside a map), or by handing whole fields to whichever field manager last wrote them (the field manager is the name Kubernetes records for the component that most recently set a given field).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas          # the HPA owns replica count, not Git
    - group: ""                    # core API group
      kind: Secret
      jqPathExpressions:
        - '.data["ca.crt"]'        # a controller re-writes this field
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - kube-controller-manager  # defer to fields this manager owns

Keep that list short and specific. Every path you tell Argo to ignore is a path where an attacker's change stops showing up as drift. Ignore /spec/template/spec/containers and you have blinded the tripwire to image swaps and injected containers, the exact changes you built it to catch. Scope each entry to one field on one kind, leave a comment naming the controller that owns it, and re-check the list every time you add a webhook or controller. Good habit: run argocd app diff after every deploy and confirm the only thing it ignores is the field you meant to ignore.

Try this

Run argocd app get guestbook on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a refresh is not a sync. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related