Hardening the Atlantis server

It holds the keys to your infra.

A hotel concierge holds the master key to every room and acts on notes slipped under the office door. Your Atlantis server is that concierge. It keeps live cloud credentials warm, and it runs terraform plan and apply the moment a webhook lands from your Git host. Hardening it is the same job you would do for the concierge. Check that each note really came from a registered guest (webhook authentication plus the repository allowlist). Limit which tasks the concierge will perform no matter what a note says (server-side workflows). Swap the master key for single-room keys cut on demand (short-lived, scoped cloud roles).

Those are the three layers: inbound, execution, credentials. A weakness in any one exposes the other two. And one fact sharpens the whole exercise. terraform plan on its own executes code. Providers are ordinary binaries that Terraform downloads and runs, and the external data source runs whatever program the configuration names. So an attacker who can get Atlantis to *plan* a malicious diff already has code execution with the server's credentials. No apply needed. Every gate below exists because "they can't apply" is not a security boundary.

Shrink what the network can reach

Atlantis puts two doors on the network, and they have very different visitor lists. The events endpoint (/events) has to be reachable by your VCS (version control system, meaning GitHub, GitLab or Bitbucket) so webhooks can land. The web interface, which is everything else (lock listing, plan logs, discard buttons), should be reachable by almost nobody, because plan output routinely spills resource names, IP addresses and the occasional secret. Terminate TLS (transport layer security, the encryption behind the padlock in your browser) either at the server with --ssl-cert-file/--ssl-key-file or at an ingress in front of it. Turn on basic auth for the interface. Leave the JSON API off; it only wakes up when you set --api-secret, and a knob you never configure is attack surface you never carry. Every flag has a matching ATLANTIS_* environment variable, which is the tidiest way to ship this config:

# /etc/atlantis/atlantis.env — every flag maps to an env var:
# --repo-allowlist -> ATLANTIS_REPO_ALLOWLIST
ATLANTIS_ATLANTIS_URL=https://atlantis.internal.example.com
ATLANTIS_REPO_ALLOWLIST=github.com/acme-corp/*   # exact org scope; never "*"
ATLANTIS_GH_USER=acme-atlantis-bot
ATLANTIS_GH_TOKEN=<injected from secret manager, never stored in this file>
ATLANTIS_GH_WEBHOOK_SECRET=<injected from secret manager>
 
ATLANTIS_WEB_BASIC_AUTH=true                      # gate the UI
ATLANTIS_WEB_USERNAME=admin
ATLANTIS_WEB_PASSWORD=<long random value>
ATLANTIS_SSL_CERT_FILE=/etc/atlantis/tls/tls.crt
ATLANTIS_SSL_KEY_FILE=/etc/atlantis/tls/tls.key
 
ATLANTIS_REPO_CONFIG=/etc/atlantis/repos.yaml     # server-side config (trusted)
ATLANTIS_ALLOW_FORK_PRS=false                     # the default — keep it
ATLANTIS_ALLOW_COMMANDS=plan,apply,unlock         # drop commands you don't use
ATLANTIS_WEBSOCKET_CHECK_ORIGIN=true              # log-stream CSRF guard
ATLANTIS_HIDE_PREV_PLAN_COMMENTS=true             # less plan detail lingering in PRs

If a load balancer or ingress sits in front of Atlantis, split the traffic by path. Expose /events to the published webhook address ranges of your VCS (gh api meta --jq .hooks prints GitHub's) and keep every other path internal. With GitHub.com the webhook has to cross the public internet. A self-hosted GitLab or Bitbucket can keep the whole exchange on private networks.

Prove every webhook is genuine

The webhook secret works like a wax seal on a letter. It is a shared key your VCS uses to compute an HMAC (hash-based message authentication code, a short fingerprint only someone holding the key can produce) over each payload, sent as the X-Hub-Signature-256 header on GitHub. Atlantis recomputes that fingerprint and compares it before parsing a single field. Without it, anyone who can reach /events can forge pull request comments, atlantis apply included. The secret proves *who sent* the event. The repo allowlist decides *which repositories* are allowed to drive the server at all. You want both. A perfectly signed webhook from a repository outside the allowlist gets dropped, and an allowlisted repository name stuffed into a forged payload fails the signature check.

# 1. Generate a strong secret
WEBHOOK_SECRET=$(openssl rand -hex 32)
 
# 2. Create one org-level webhook covering every repo in the org
gh api orgs/acme-corp/hooks -f name=web \
  -F active=true \
  -f 'config[url]=https://atlantis.internal.example.com/events' \
  -f 'config[content_type]=json' \
  -f "config[secret]=$WEBHOOK_SECRET" \
  -f 'events[]=pull_request' -f 'events[]=push' \
  -f 'events[]=issue_comment' -f 'events[]=pull_request_review'
# {
#   "id": 4931072,
#   "active": true,
#   "events": ["pull_request", "push", "issue_comment", "pull_request_review"],
#   "config": { "url": "https://atlantis.internal.example.com/events", ... }
# }
 
# 3. Prove unsigned requests bounce off
curl -si -X POST https://atlantis.internal.example.com/events \
  -H 'X-GitHub-Event: ping' -H 'Content-Type: application/json' -d '{}'
# HTTP/2 400
# missing signature

Troubleshooting runs both ways. A burst of 400s in the Atlantis log right after a secret rotation means the VCS and the server no longer agree, so rotate both sides inside one change window. And prefer running Atlantis as a GitHub App over per-repo hooks driven by a personal access token. The app brings its own webhook with the secret built in, its permissions are scoped to exactly what Atlantis needs, and its installation tokens expire every hour. The *Credentials & secrets* lesson covers that side in depth.

Where trust stops: fencing in execution

Recall the split from the server-side config lesson. repos.yaml lives on the server and is trusted. atlantis.yaml lives in the repository and is attacker-controlled the moment an outsider can open a pull request. The most dangerous thing you can hand to that file is the custom run step, which executes arbitrary shell as the Atlantis user, with the Atlantis credentials, during plan. Hardening means the server keeps that power. Define workflows centrally, set allowed_overrides to nothing (or to workflow alone, so repositories may pick among *your* predefined workflows), and refuse repo-defined workflows outright.

# /etc/atlantis/repos.yaml — server-side, trusted
repos:
  - id: /github\.com\/acme-corp\/.*/
    branch: /^main$/
    apply_requirements: [approved, mergeable, undiverged]
    allowed_overrides: []          # repo atlantis.yaml may override nothing
    allow_custom_workflows: false  # no repo-defined run steps -> no PR-driven RCE
    workflow: default
 
workflows:
  default:
    plan:
      steps: [init, plan]
    apply:
      steps: [apply]

The apply requirements turn code review into your change-control gate. approved demands a reviewer's sign-off before atlantis apply will run. mergeable demands that branch protection passes. undiverged blocks an apply from a branch that has fallen behind its base. Keep --allow-fork-prs at its default of false. Fork pull requests are strangers' code, and as established above, planning strangers' code *is* running strangers' code.

Ship it hardened on Kubernetes

The official Helm chart (runatlantis/atlantis) deploys a StatefulSet rather than a Deployment. Atlantis keeps working plans and its lock database in a data directory, so it needs a PersistentVolumeClaim (a slice of disk that survives restarts) and a stable pod identity. The hardening sits on top of that. Feed VCS credentials from a Kubernetes Secret you created yourself instead of pasting them into chart values. Attach cloud permissions through workload identity (IRSA, short for IAM Roles for Service Accounts, on Amazon EKS; Workload Identity on Google GKE) so no static cloud keys exist anywhere. Run as the image's non-root atlantis user. Allowlist GitHub's webhook address ranges at the ingress.

orgAllowlist: github.com/acme-corp/*
github:
  user: acme-atlantis-bot
vcsSecretName: atlantis-vcs        # existing Secret: keys github_token + github_secret
 
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/atlantis  # IRSA: no static keys
 
statefulSet:
  securityContext:
    runAsNonRoot: true
    runAsUser: 100                 # the image's "atlantis" user
    fsGroup: 1000
 
environment:
  ATLANTIS_WEB_BASIC_AUTH: "true" # username/password via environmentSecrets, not values
  ATLANTIS_WEBSOCKET_CHECK_ORIGIN: "true"
  ATLANTIS_HIDE_PREV_PLAN_COMMENTS: "true"
 
ingress:
  enabled: true
  host: atlantis.example.com
  annotations:
    # only GitHub's hook ranges may connect: gh api meta --jq .hooks
    nginx.ingress.kubernetes.io/whitelist-source-range: "192.30.252.0/22,185.199.108.0/22,140.82.112.0/20,143.55.64.0/20"
  tls:
    - secretName: atlantis-tls
      hosts: [atlantis.example.com]
 
resources:
  requests: {cpu: 500m, memory: 1Gi}
  limits: {memory: 4Gi}

helm repo add runatlantis https://runatlantis.github.io/helm-charts
helm repo update
 
kubectl create namespace atlantis
kubectl -n atlantis create secret generic atlantis-vcs \
  --from-literal=github_token="$GH_TOKEN" \
  --from-literal=github_secret="$WEBHOOK_SECRET"
 
helm upgrade --install atlantis runatlantis/atlantis -n atlantis -f values.yaml
# Release "atlantis" does not exist. Installing it now.
# NAME: atlantis
# NAMESPACE: atlantis
# STATUS: deployed
 
kubectl -n atlantis get statefulset,pvc
# NAME                        READY   AGE
# statefulset.apps/atlantis   1/1     52s
# NAME                                             STATUS   CAPACITY
# persistentvolumeclaim/atlantis-data-atlantis-0   Bound    5Gi
 
curl -s https://atlantis.example.com/healthz
# {
#   "status": "ok"
# }

Finish with a NetworkPolicy, which is the cluster's door policy for a single pod. Inbound traffic only from the ingress controller's namespace. Outbound traffic only to DNS (domain name system, the internet's phone book), your VCS and your cloud provider's APIs. It will not neutralize the credentials the pod legitimately holds. What it does is turn "an attacker got code execution in the pod" into a contained incident rather than a launch pad for moving sideways across your cluster network.

Verify, monitor, patch

Treat the server like the security system it is. Re-verify each control after every deploy; those curl probes above belong in your smoke tests. Then actually read the logs. Every plan and every apply is a security event with an actor, a repository and a diff, which makes Atlantis logs excellent audit input for a SIEM (security information and event management system, the tool your security team searches). Alert on three patterns. 400s on /events mean signature mismatches, so either a rotation went wrong or somebody is probing. Plans against repositories you do not recognize mean your allowlist is too wide. atlantis unlock usage means somebody is force-releasing the locks that keep changes to a directory serialized. Patch on a schedule too. Atlantis ships releases often, those releases regularly bump bundled Terraform and dependency versions, and pinning the container image by digest keeps every deploy reproducible.

Every gate here is work you own: certificates, secret rotation, chart upgrades, log pipelines, patch windows. That operational bill is the honest price of self-hosting the pull-request automation loop, and it is exactly the axis the hosted platforms compete on. Next, *Atlantis vs the alternatives* weighs that trade.

Terraform's external data source and provider plugins are arbitrary code running during plan. That is why network egress controls on the Atlantis pod matter, and why admission policies that stop extra secrets being mounted into the Atlantis namespace matter too. A soft interior behind hard-looking YAML is how concierge desks get robbed.

Split prod and non-prod Atlantis wherever blast radius demands it. One bot with org-wide apply rights is convenient and catastrophic in the same breath. Prefer a role per environment and an allowlist per environment, even when that means running two deployments.

Add an egress policy that lets Atlantis reach your VCS, your provider APIs and your module sources, and nothing else. A plan that tries to pull a random script off a paste site should fail closed. Pair that with a read-only root filesystem where the chart allows it, and keep the data volume the only writable path.

Try this

Walk the three layers on your own install: webhook authentication plus allowlist (inbound), server-side workflows (execution), scoped cloud roles (credentials). Write down one gap per layer.

curl -sI https://atlantis.example.com/healthz
# confirm TLS, no public UI without auth if exposed
grep -E "allowlist|workflow|apply" /etc/atlantis/repos.yaml | head -20

HTTP/2 200
# repos.yaml: allowlist: github.com/acme/infra-*
# apply_requirements: [approved, mergeable]
# allow_custom_workflows: false

Takeaway

The rule to carry: harden inbound (webhook secret, allowlist), execution (server-side workflows, no arbitrary run steps from pull requests) and credentials (scoped, short-lived). Plan executes code, so treat plan access the way you treat production access.

Next, close the biggest gap you wrote down, add delivery monitoring for your webhooks, and book a tabletop exercise where somebody tries to escalate through a malicious Terraform external data source.

Related