>_SecOpsLog_
CoursesArgo CDSync waves & hooks

Sync waves & hooks

Order and lifecycle jobs.

Advanced12 min · lesson 6 of 12

A kitchen does not send every dish out the moment the order comes in. The stock simmers before the sauce is built on it, the oven heats before the bread goes in, and the plate is wiped before the food lands. Kubernetes applications carry the same kind of ordering. A CRD (Custom Resource Definition, the way you teach Kubernetes a brand-new kind of object) has to exist before anything uses it. A database schema has to be migrated before the new code that expects new columns starts serving. A namespace (a named partition inside a cluster) has to exist before you put anything in it. Plain 'apply everything at once' knows none of this. Argo CD gives you two controls for order and timing inside a sync, its word for making the live cluster match what you committed to Git: sync waves and hooks.

Waves: Board the Plane by Group Number

Airlines board by group number. Group 1 first, then 2, then 3, and nobody in group 3 walks down the jet bridge until group 2 is aboard. A sync wave is that group number for your resources. You attach an annotation (a small key-value tag on a resource) called argocd.argoproj.io/sync-wave with an integer, and Argo CD applies the lowest numbers first. Everything without the annotation sits at wave 0. Negative numbers run before 0, positive numbers after. So you might put a CRD at wave -2, its configuration at -1, the workload that uses them at 0, and a smoke test at 1. One 'sync' then rolls the whole thing out in the right sequence instead of firing every manifest (each YAML file that describes one Kubernetes object) at the cluster in one shot.

manifests.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: shop-config
  namespace: shop
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
data:
  LOG_LEVEL: info
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
  namespace: shop
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # default, written out for clarity
spec:
  replicas: 3
  # ...pod template that reads shop-config...

Here is what Argo CD actually does with those numbers. It sorts every resource into waves, applies all the resources in the lowest wave, and then waits. It will not start the next wave until every resource in the current one reports Healthy. A Deployment (a controller that keeps a set of identical pods, the smallest runnable unit in Kubernetes, running) is Healthy once its pods are up and ready; a Job is Healthy once it finishes successfully. Between waves the application controller (the Argo CD component that performs the sync) pauses for a short delay, set by the environment variable ARGOCD_SYNC_WAVE_DELAY on the controller, two seconds by default. Two things follow. First, ordering inside a single wave is decided for you by resource kind and then name, so namespaces and CRDs already land before the workloads that need them; you reach for waves when the obvious kind order is not enough. Second, every wave is a health-gated stop, so a manifest with ten waves is a slower, more serial sync than one with two. Use waves where order truly matters, not as decoration.

Hooks: Pre-Flight and Post-Flight Checks

Waves order the things that stay. Hooks handle the things that run once and are then done. A pilot runs a pre-flight checklist before pushing back and a walkaround after landing; neither is part of the aircraft, both are tied to a moment in the trip. A hook is a resource, almost always a Kubernetes Job (a one-shot task that runs a pod to completion), that Argo CD runs at a chosen phase of the sync. You mark it with the argocd.argoproj.io/hook annotation. The phases are PreSync, Sync, PostSync, and SyncFail.

db-migrate.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "0"
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: db-migrator
      containers:
        - name: migrate
          image: registry.example.com/shop-api:1.4.0
          command: ["/app/migrate", "up"]

PreSync runs before any normal resource is applied, which is exactly where a database migration belongs: change the schema, then roll the code that needs it. Sync runs alongside your ordinary resources, in the same wave order. PostSync runs only after every Sync resource is Healthy, which is where a smoke test or a cache warm belongs. SyncFail runs only when the sync fails, so it is your place for cleanup or a rollback trigger. There is also a Skip value, which tells Argo CD not to apply a resource at all. Hooks obey waves too, so a PreSync hook at wave -1 runs before a PreSync hook at wave 0. When a hook finishes, the argocd.argoproj.io/hook-delete-policy annotation decides its fate: HookSucceeded deletes it once it succeeds, HookFailed deletes it once it fails, and BeforeHookCreation (the default when you set nothing) keeps it around and removes the old copy only just before the next sync recreates it. That default is handy, because it leaves the finished Job in place for you to read its logs.

Fixed-Named Hook Jobs Can Collide
A Job's pod template is immutable once the Job exists. If a hook uses a fixed metadata.name and a delete policy that keeps it around, the next sync tries to re-apply a changed Job and Kubernetes rejects the update. Either rely on BeforeHookCreation to remove the old copy before creating the new one, or give the hook metadata.generateName instead of metadata.name so every run gets a fresh, uniquely named Job and a full history.

Watch a Sync Run

terminal
argocd app sync shop
output
Name:               argocd/shop
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          shop
URL:                https://argocd.example.com/applications/shop
Repo:               https://github.com/acme/shop-manifests
Target:             HEAD
Path:               overlays/prod
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        Synced to HEAD (9f3c1ab)
Health Status:      Healthy
 
Operation:          Sync
Sync Revision:      9f3c1ab2c0e5d1f4a8b7c6d5e4f3a2b1c0d9e8f7
Phase:              Succeeded
Start:              2026-07-20 09:14:02 +0000 UTC
Finished:           2026-07-20 09:14:41 +0000 UTC
Duration:           39s
Message:            successfully synced (all tasks run)
 
GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK      MESSAGE
       Namespace   shop       shop          Synced            namespace/shop created
       ConfigMap   shop       shop-config   Synced            configmap/shop-config created
batch  Job         shop       db-migrate    Synced  Healthy  PreSync   job.batch/db-migrate created
       Service     shop       shop-api      Synced  Healthy            service/shop-api created
apps   Deployment  shop       shop-api      Synced  Healthy            deployment.apps/shop-api created
batch  Job         shop       smoke-test    Synced  Healthy  PostSync  job.batch/smoke-test created

Read that output top to bottom and you can watch the plan execute. The db-migrate Job carries the PreSync marker in the HOOK column and lands before the Service and Deployment. The Namespace and ConfigMap go in ahead of the workload because of their negative waves. The smoke-test Job runs last with the PostSync marker. The final Message, 'successfully synced (all tasks run)', means every phase and every hook completed. If the PreSync Job had failed, the Phase would read Failed and the Deployment lines would never appear.

terminal
kubectl -n shop get jobs
output
NAME         COMPLETIONS   DURATION   AGE
smoke-test   1/1           5s         2m

Notice db-migrate is gone. Its delete policy was HookSucceeded, so Argo CD removed it the moment it finished. The smoke-test Job set no delete policy, so it stays, which is why you can still read its logs. If you want to keep migration output for a post-mortem, drop the delete policy on that Job and it will persist until the next sync recreates it.

terminal
kubectl -n shop logs job/smoke-test
output
+ curl -fsS http://shop-api.shop.svc.cluster.local:8080/healthz
{"status":"ok","version":"1.4.0"}
+ curl -fsS http://shop-api.shop.svc.cluster.local:8080/ready
{"ready":true,"db":"connected"}
smoke test passed

The Security Tradeoff You Take On

A hook is a resource in your Git repository, and Argo CD's application controller applies it with the controller's own reach into the cluster. That makes a hook a way to run code inside your cluster on every sync. Anyone who can merge to the manifests repository, or who steals a token that can, is able to add a PreSync Job that pulls any image and runs any command, under whatever ServiceAccount (the identity Kubernetes hands a pod so it can talk to the cluster's API, its control interface) the Job names. It runs before the application even appears, before a human looks at the new version, as an ordinary step of 'deploy'. That is a quiet, well-placed foothold.

So treat the manifests repo as production, because a merge to it runs code in production. Require review on pull requests that touch hook Jobs, and actually read the image and command each one runs. Fence every app with an AppProject (an Argo CD object that limits which clusters, namespaces, and resource kinds an application may touch) so a hook cannot reach outside its lane. Give hooks a dedicated, low-privilege ServiceAccount instead of the namespace default, and set automountServiceAccountToken: false when the Job needs no API access at all. Then watch the Jobs themselves, with RBAC (Role-Based Access Control) locked down: alert on hook Jobs whose image is not from your registry, and on hook failures, since a wedged PreSync is both an outage and, now and then, a sign of tampering.

A Failing PreSync Hook Wedges Every Deploy
PreSync gates the rollout on purpose, so a hook that fails stops the new version cold: nothing else in the sync runs until the hook passes. A flaky or non-idempotent migration Job therefore sits on the critical path of every deploy. Make hook Jobs idempotent (safe to run a second time with no harm), give them a backoffLimit and an activeDeadlineSeconds timeout so a stuck one fails instead of hanging, and test the failure path, not only the happy one.
How one sync executes
1PreSync
hooks first (e.g. DB migration); a failure aborts the whole sync
2Sync
resources apply wave by wave, lowest number first
3Health gate
each wave must report Healthy before the next starts
4PostSync
hooks after all Sync resources are Healthy (e.g. smoke test)
5SyncFail
runs only if any phase failed (cleanup or rollback)

To confirm your ordering actually holds, do not trust the manifest, watch a real sync. Run argocd app sync and read the HOOK column and the order of the resource lines, or open the same view in the web UI where each wave and phase is drawn as a separate step. After it finishes, argocd app get shows the last operation's phase and message. If a resource applied in the wrong order, its wave annotation is missing or wrong, and the sync output is where you will see it.

Quick check
01A PreSync hook Job exhausts its backoffLimit and fails. What happens to the rest of that sync?
Incorrect — PreSync is a gate; its failure stops the sync before the Sync phase starts.
Correct — A failed PreSync aborts the operation, so no normal resources apply and SyncFail hooks run.
Incorrect — The failure aborts the whole operation, not a single wave.
Incorrect — The operation fails and stops; it does not auto-retry unless you set syncPolicy.retry.
02You want a finished hook Job to stay in the cluster after it succeeds so you can still read its logs, and be removed only just before the next sync recreates it. Which value of the argocd.argoproj.io/hook-delete-policy annotation does that?
Incorrect — HookSucceeded deletes the Job the moment it succeeds, so its logs are gone before you can read them.
Incorrect — HookFailed only removes the Job when it fails; it does not describe the keep-until-next-sync behavior you want for a successful hook.
Correct — BeforeHookCreation is the default when you set nothing; it leaves the finished Job in place and removes the old copy only just before the next sync recreates it.
Incorrect — Skip is not a delete policy at all; it is a hook value that tells Argo CD not to apply a resource in the first place.
03Your manifests include a database-migration Job annotated argocd.argoproj.io/hook: PreSync with sync-wave "0", and a Deployment that has no hook annotation and sits at the default wave 0. During one sync, which applies first and why?
Correct — phase ordering gates before wave ordering; PreSync runs before any normal resource, so the shared wave 0 does not put the Deployment ahead.
Incorrect — a PreSync hook is not an ordinary wave 0 resource; the whole PreSync phase precedes the Sync phase.
Incorrect — the Deployment is a Sync-phase resource and the Job is a PreSync hook, so the phase split orders them rather than applying them together.
Incorrect — a PreSync hook runs once, before the Sync phase; Argo CD does not move it to PostSync.

Before you put a PreSync hook in front of your rollout, run its container by hand against a scratch database twice in a row. If the second run is clean, the Job is idempotent and safe to sit on your critical path. If it is not, fix that first, because Argo CD will run it there for you at two in the morning whether it is ready or not.

Try this

Run argocd app sync shop on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: fixed-Named Hook Jobs Can Collide. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Sync policies & self-healApp-of-apps & ApplicationSets

Related

What Atlantis is: PR-driven TerraformAtlantisWhat Flux is: the GitOps toolkitFluxArgo CD cheat sheetCheat sheetArgo CD vs FluxComparisonGitOps interview questionsInterview prep