Your pipeline builds a fresh image, the bump job pushes to the state repo, and Argo CD reports Synced / Healthy. The new code still is not live. What is the most likely explanation?

The bump job wrote the mutable tag :latest, so the manifest text was identical and the agent had nothing to reconcile. A no-op diff. The YAML text never moved, so the agent reports Synced against a manifest still pointing at the old bits. Pin the immutable digest and every build becomes a real change the agent can act on.

In pull-based GitOps, what is the single biggest security change compared with a push-based pipeline that runs `kubectl apply`?

Standing cluster credentials leave CI entirely, so a leaked runner token can no longer reach the API server, because only the in-cluster agent touches the cluster. Taking cluster credentials out of CI is the largest cut to blast radius available; the most a compromised pipeline manages is a commit that review and admission still gate.

The payments Argo CD Application runs `syncPolicy.automated` with `selfHeal: true`. Mid-incident, an on-call engineer edits the live Deployment directly with `kubectl` to change a setting. What does the agent do?

It reverts the live change back to what Git declares, within the next reconcile cycle. SelfHeal keeps correcting drift so the running cluster matches the committed state.

Courses Secure CI/CD with GitLabGitOps with Argo CD / Flux

GitOps with Argo CD / Flux

Git as the source of truth for deploys.

Advanced14 min · lesson 15 of 17

Your pipeline just built payments:2.7.0, scanned it clean, and cosign-signed the digest sha256:9b2c1e...f0 (a digest is a fingerprint of the exact image bytes; unlike a tag, nobody can move it to point at something else). One job is left: get that image running in production. The usual answer hands the CI (continuous integration) job a kubeconfig, the credential file that opens your Kubernetes cluster, and lets it run kubectl apply or helm upgrade. The pipeline reaches in and pushes. GitOps reverses that arrow, and a thermostat is the closest everyday match. You set the temperature you want on the dial, and a small box on the wall keeps checking the actual room and closing the gap, forever. In GitOps the dial is a Git repository, the state repo, holding declarative YAML (plain text files that describe what should exist) for every Deployment, Service, and image digest production is supposed to have. The box on the wall is an agent running inside the cluster, either Argo CD or Flux, comparing that repo against the live cluster and correcting whatever does not match. You never push to the cluster. You commit to Git, and the agent pulls. That single inversion, pull instead of push, is what this lesson builds and then verifies.

How a new image reaches the cluster

How does a new image reach the cluster?

the last step of every pipeline

push CD

CI runs kubectl / helm

the pipeline holds a kubeconfig and applies directly: few moving parts, works anywhere, but a leaked runner token reaches the API server

pull GitOps

CI commits a digest bump

an in-cluster Argo CD / Flux agent pulls the state repo and reconciles, so no cluster credentials live in CI at all

Push makes the pipeline the actor. Pull makes Git the source of truth, and the agent the only thing that touches the cluster.

Push or pull, and how to pick one

Push-based CD (continuous delivery) is where most teams start. The pipeline is the actor. It holds the cluster credential, and it runs the deploy command. Few moving parts, and it works against anything: virtual machines, serverless functions, Kubernetes. What it costs you is trust. Every runner (the machine or container that executes a pipeline job) able to pick up the deploy job sits one leaked token away from your API server, the front door of the cluster, and the live state of production exists only as whatever the last apply happened to set. There is no file to compare it against. Pull-based GitOps adds one component, the in-cluster agent, and gives up some reach along the way: it only handles declarative targets, which in practice means Kubernetes. Three things come back in exchange. CI loses every scrap of cluster access. Live state becomes a file you can read, diff and revert. And drift, meaning the running cluster quietly wandering away from what the files say, gets corrected without anyone filing a ticket. For a fleet of Kubernetes clusters, that trade is nearly always worth making. For one lonely VM, keep pushing.

CI's last job edits one line of text

Be exact about the handoff. The final job in your pipeline never speaks to Kubernetes at all. It changes one line in the state repo. Using yq (a command-line editor for YAML files), or kustomize edit set image if you prefer, it rewrites the container image to the precise digest this pipeline produced, commits, and pushes. The credential it carries is not a kubeconfig. It is a Project Access Token, a GitLab token tied to one project, scoped to write_repository on the deploy repo and nothing else, delivered to the job as a masked, protected CI/CD variable. If an attacker owns your pipeline, the most they get is a commit, and that commit still has to survive human review and admission control, the gatekeeper that inspects every object before the cluster accepts it. They cannot kubectl anything.

.gitlab-ci.yml

bump-manifest:
  stage: deploy
  image: alpine:3.20
  needs: ["build-sign"]                 # consumes IMAGE, DIGEST from build.env (dotenv)
  variables:
    DEPLOY_REPO: gitlab.acme.internal/acme/deploy.git
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - apk add --no-cache git yq
    - git clone "https://bump-bot:${DEPLOY_TOKEN}@${DEPLOY_REPO}" state
    - cd state
    - yq -i '.spec.template.spec.containers[0].image = strenv(IMAGE) + "@" + strenv(DIGEST)' payments/deployment.yaml
    - git -c [email protected] -c user.name=ci-bump-bot commit -am "payments -> ${DIGEST} (${CI_COMMIT_SHORT_SHA})"
    - git push origin main

IMAGE and DIGEST arrive as environment variables from the build stage's dotenv artifact (artifacts:reports:dotenv: build.env), a small file of KEY=value lines that later jobs load automatically, which is why yq's strenv() can read them directly. DEPLOY_TOKEN is protected, so GitLab hands it out only to pipelines running on protected branches. A feature branch never sees it. A pipeline from a forked project's merge request (MR) never sees it either, so neither one can reach the state repo. Read the job log below as a receipt: the commit hash and the successful push are the written record of what production is about to become.

job log — bump-manifest

$ yq -i '.spec.template.spec.containers[0].image = strenv(IMAGE) + "@" + strenv(DIGEST)' payments/deployment.yaml
$ git -c [email protected] -c user.name=ci-bump-bot commit -am "payments -> ${DIGEST} (${CI_COMMIT_SHORT_SHA})"
[main 3f9a1c2] payments -> sha256:9b2c1e...f0 (a1b2c3d)
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git push origin main
To https://gitlab.acme.internal/acme/deploy.git
   7e4d2a1..3f9a1c2  main -> main
Cleaning up project directory and file based variables
Job succeeded

Watching the agent do its round

On the cluster side, an Argo CD Application object says where the desired state lives and how hard to enforce it. syncPolicy.automated with prune and selfHeal reads as three standing promises: apply new commits without being asked, delete objects that were removed from Git, and drag any live drift back to what Git declares. Left alone, Argo CD rechecks the repo every couple of minutes (timeout.reconciliation defaults to 120s, plus up to 60s of jitter so a thousand apps do not all poll on the same tick). A GitLab push webhook cuts that wait down to seconds, and argocd app sync forces it right now. In the terminal below, --refresh shows the app OutOfSync at the new commit, the sync applies it, and the final get confirms both the synced revision and the digest actually running.

payments.yaml (Argo CD Application)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: payments, namespace: argocd }
spec:
  source:
    repoURL: https://gitlab.acme.internal/acme/deploy.git
    path: payments
    targetRevision: main
  destination: { server: https://kubernetes.default.svc, namespace: payments }
  syncPolicy:
    automated: { prune: true, selfHeal: true }   # revert drift, delete removed objects

terminal — argocd

$ argocd app get payments --refresh
Name:            argocd/payments
Sync Policy:     Automated (Prune, SelfHeal)
Sync Status:     OutOfSync from main (3f9a1c2)
Health Status:   Healthy
 
$ argocd app sync payments
Operation:       Sync
Sync Revision:   3f9a1c2c9e7b1a4d...
Phase:           Succeeded
Message:         successfully synced (1 task run)
 
$ argocd app get payments
Sync Status:     Synced to main (3f9a1c2)
Health Status:   Healthy
Images:          registry.acme.internal/payments@sha256:9b2c1e...f0

Flux builds the same loop out of two pieces, a GitRepository source that tracks the repo and a Kustomization that applies what it finds, reconciling on a timer or whenever you ask. flux reconcile kustomization payments --with-source fetches the new commit and applies it in one move, printing the revision it settled on. That printed revision is your proof that the cluster now matches Git.

terminal — flux

$ flux reconcile kustomization payments --with-source
► annotating GitRepository payments in flux-system namespace
✔ GitRepository reconciliation completed
✔ fetched revision main@sha1:3f9a1c2
► annotating Kustomization payments in flux-system namespace
✔ Kustomization reconciliation completed
✔ applied revision main@sha1:3f9a1c2

Why pull is the safer default

The gains here come from the shape of the system rather than from anything bolted on afterwards. First, standing cluster credentials leave CI completely. That is the biggest cut to blast radius on offer, because a leaked runner token or a poisoned build dependency no longer has a route to the API server; the most it manages is a commit, with review and admission control still standing in the way. Second, every production change is now a Git commit with an author, a timestamp and a reviewer, so the repo doubles as your audit log and your rollback button. Third, selfHeal turns drift into a problem that fixes itself: hand-edit a live Deployment and the agent puts it back within a cycle. Notice what moved, though. The question stopped being 'who holds cluster credentials' and became 'who can merge to the state repo'. That repo is production now. Guard it like production: protected branch, a CODEOWNERS file naming the reviewers each path requires, required approvals, signed commits, and an agent whose RBAC (role-based access control, the rules deciding what it may touch) covers only the namespaces it owns.

Rollback comes free out of the same machinery. The deployed digest is a line of text in Git, so reverting the bump commit is the rollback. The agent spots the change and pulls the cluster back to the previous known-good digest within a cycle, no special tooling, nobody holding a kubeconfig at 3am. 'How fast can we get back to the last good version?' now has a dull, dependable answer: git revert, then let the loop run. What is still missing is a gate in front of that merge, because in GitOps the merge to the state repo is the deploy. The next lesson builds it.

In production: tokens, forks, and fifty services

Two realities bend this handoff once a real team is using it. Start with least privilege and forks. DEPLOY_TOKEN has to be masked and protected, so GitLab exposes it only on protected branches. A fork MR pipeline or a feature-branch pipeline never receives the value at all, cannot clone the state repo with it, and cannot echo it into a log. That shuts the classic fork trick, where an outsider opens a merge request whose real purpose is to print your secrets. Then scale. One Application per service is comfortable at five services and unworkable at fifty, so teams generate them with ApplicationSets, or with an app-of-apps root Application that lives in Git and gets reconciled exactly like the rest. Neither move changes the security model. They keep the pull loop quick, least-privileged and manageable as the service count climbs.

A mutable tag turns reconcile into a silent no-op

If the bump job writes a mutable tag such as image: payments:latest instead of the immutable digest, the text of the manifest does not move when you build a new payments:latest. Argo CD and Flux compare text. They see nothing different, so they do nothing. The pipeline goes green, the agent reports Synced, and the old bits keep serving real traffic. Always pin the digest (...@sha256:...). Kubernetes re-pulls an image only when the pod spec changes, the digest is the thing that changes it, and the digest is also what your signature and your admission policy check against.

Try this

Run argocd app get payments --refresh on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a mutable tag turns reconcile into a silent no-op. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

SBOM & provenance attestationsDeploy gates & approvals

What CI is, and where Jenkins fitsJenkins foundations, done right The software supply chain & why it is targetedSoftware supply chain security Supply-chain threat modelSoftware supply chain in depth Git cheat sheetCheat sheet GitLab CI/CD cheat sheetCheat sheet CI/CD interview questionsInterview prep