Credentials & secrets

Cloud creds without leaking them.

A hotel valet stand is a good trade for everyone. Guests stop circling the block, nobody scrapes a bumper on the ramp, and the garage runs on one set of rules. The catch is the pegboard behind the desk. It now holds a key to every car on the lot. Atlantis makes that same trade with your infrastructure. Credentials come off dozens of engineer laptops and land on one server, and that server takes its orders from pull requests that strangers can open. Your job is to make the pegboard hold as little as possible, for as short a time as possible.

Five secrets, one server

Start with an inventory, because you cannot protect what you never wrote down. A running Atlantis handles five kinds of secret material. First, the cloud credentials Terraform uses to build infrastructure. Second, the VCS token (version control system, meaning your GitHub or GitLab API credential), which Atlantis uses to clone repos, post plan comments, and set the commit statuses your merge rules check. Third, the webhook secret, a random string shared with your Git host so it can prove a delivery genuinely came from them. Fourth, any secret Terraform variables a plan needs as input, like a database password or a provider API key. Fifth, the artifacts Terraform leaves behind: state files and saved plan files, both of which can hold secrets in readable text. Each one attracts a different attacker and needs a different fix, so take them in order.

Cloud credentials: assume, don't store

The worst version of this is a static AWS access key pair sitting in the server's environment. It never expires, it usually carries broad power, and it lives on the most exposed machine you run. The better version is workload identity, where the platform hands the running pod a short-lived identity and no permanent key exists anywhere at all. It works like a visitor badge the front desk reprints every few hours, rather than a spare key you cut once and then forget about. On EKS (Elastic Kubernetes Service, Amazon's managed Kubernetes) that mechanism is IRSA (IAM Roles for Service Accounts). The cluster's OIDC provider (OpenID Connect, a standard way for one system to vouch for an identity to another) signs a token for the pod's ServiceAccount, and AWS STS (Security Token Service) trades that token for temporary credentials on a role. GKE Workload Identity and Azure federated credentials do the same dance. From that *base* role, each Terraform project then assumes a *scoped* per-environment role using assume_role in its provider block, so the staging project physically cannot reach prod. One built-in limit is worth learning before it bites you. That second hop is role chaining, and AWS caps a chained session at one hour no matter what maximum you configured on the role. A very long apply needs either a role the pod can assume directly, or a project broken into smaller pieces.

orgAllowlist: github.com/acme-corp/*
 
github:
  user: atlantis-bot          # token + webhook secret come from the Secret below
 
vcsSecretName: atlantis-vcs   # pre-created Secret; keys: github_token, github_secret
 
serviceAccount:
  create: true
  annotations:
    # IRSA: the pod gets short-lived creds for this role — no AWS keys anywhere
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/atlantis-base
 
volumeClaim:
  enabled: true
  dataStorage: 8Gi            # holds repo clones + plan files — treat as secret storage

# 1. Token + webhook secret live in a k8s Secret — never in values.yaml or git
kubectl create ns atlantis
kubectl -n atlantis create secret generic atlantis-vcs \
  --from-literal=github_token='ghp_2XwR…' \
  --from-literal=github_secret="$(openssl rand -hex 32)"
# secret/atlantis-vcs created
 
# 2. Deploy the official chart (it runs Atlantis as a StatefulSet)
helm repo add runatlantis https://runatlantis.github.io/helm-charts
helm upgrade --install atlantis runatlantis/atlantis -n atlantis -f values.yaml
# STATUS: deployed
# REVISION: 1
 
# 3. Prove there are no static cloud keys in the pod — only IRSA's web identity
kubectl -n atlantis exec sts/atlantis -- env | grep ^AWS
# AWS_STS_REGIONAL_ENDPOINTS=regional
# AWS_DEFAULT_REGION=eu-west-1
# AWS_REGION=eu-west-1
# AWS_ROLE_ARN=arn:aws:iam::111122223333:role/atlantis-base
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Look at what never happens in that flow. The GitHub token and the webhook secret live in a Kubernetes Secret you created once by hand, and values.yaml only names it through vcsSecretName. The values themselves never appear in the chart, in Git history, or in helm get values output. The same rule holds on a plain virtual machine: pass secrets in as environment variables (ATLANTIS_GH_TOKEN, ATLANTIS_GH_WEBHOOK_SECRET), never as command-line flags. Anyone who can list running processes on that host can read a flag.

The VCS token and the webhook secret

The VCS token is a full write credential for your repositories. It clones private code, writes comments, and flips the commit status your branch protection rule trusts before it allows a merge. Prefer a GitHub App over a personal access token (a PAT, which is a long-lived key tied to one human's account). An App gets fine-grained permissions per installation and mints short-lived installation tokens on its own. A PAT is one engineer's durable repo-scoped credential, and it dies quietly the day that engineer leaves. The webhook secret guards traffic going the other way. Your /events endpoint has to be reachable from the internet so GitHub can call it, and with no validation, anybody who finds that URL can fake a pull-request event. The fix is an HMAC (hash-based message authentication code), which behaves like a wax seal only two parties can press: GitHub hashes every delivery payload together with the shared secret, Atlantis recomputes the same hash, and anything that does not match gets thrown out. GitLab is weaker here. Its token rides along verbatim in the X-Gitlab-Token header and gets compared as a plain string rather than used to sign anything, so TLS on the endpoint is carrying more of the load.

# Forged event, no valid signature — the webhook secret is the only gate:
curl -si -X POST https://atlantis.acme.dev/events \
  -H 'X-GitHub-Event: pull_request' \
  -H 'Content-Type: application/json' \
  -d '{"action":"opened"}' | head -n1
# HTTP/2 400
 
# Atlantis log (kubectl -n atlantis logs sts/atlantis):
# {"level":"warn","msg":"payload signature check failed"}

Terraform variables: fetch per run, redact on display

Secret *inputs*, like a database password or a Datadog API key, belong in neither the repo nor the server image. Atlantis's env step handles them. It is a custom workflow step that runs a command when the plan starts and hands that command's output to the following steps as an environment variable. Point it at your secret manager and the value exists only for the length of one run. Two rules keep this honest. Define the workflow server-side in repos.yaml. If repos are allowed to define their own workflows, anyone who can open a pull request can add a run step and execute whatever they like with every credential on this page within reach, which is the repo-side versus server-side trust boundary from the custom-workflows lesson. And give the pod its Vault access the same way you gave it cloud access: a Kubernetes-auth role, so vault logs in with the ServiceAccount token instead of a stored Vault token sitting on disk.

# Server-side config: a PR cannot edit this file.
repos:
  - id: github.com/acme-corp/infra
    workflow: prod-secrets      # pinned; 'workflow' is NOT in allowed_overrides
 
workflows:
  prod-secrets:
    plan:
      steps:
        - env:
            name: TF_VAR_db_password
            # runs per plan; stdout becomes the env var, held only for this run
            command: vault kv get -field=password secret/prod/db
        - init
        - plan

# You comment on the pull request:
atlantis plan -p prod-db
 
# Atlantis replies:
Ran Plan for project: prod-db dir: prod/db workspace: default
 
Terraform will perform the following actions:
 
  # aws_db_instance.main will be updated in-place
  ~ resource "aws_db_instance" "main" {
        id       = "prod-db"
      ~ password = (sensitive value)
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.
 
* To apply this plan, comment: atlantis apply -p prod-db
* To delete this plan and lock, click here
* To plan this project again, comment: atlantis plan -p prod-db

Two catches here. First, (sensitive value) is redaction for the screen and nothing more. It requires sensitive = true on the variable, and the saved plan file in Atlantis's data directory still holds the real password in readable text. So does state, once you apply. Treat the data-directory volume and the state backend as secret stores in their own right: encrypt them, limit who can read them, and do not drop backups somewhere easier to reach than the original. Second, that plan output is a pull-request comment, visible to everyone with read access to the repo, and it stays there. On Terraform 1.10 and later you can declare genuinely dangerous inputs ephemeral = true, and from 1.11 you pair that with a provider's write-only arguments (things like password_wo) wherever a resource has to consume the value. Then it never reaches the plan file or state at all.

When the chain breaks: troubleshooting

Credential failures show up in three different places, and telling them apart saves you an hour of guessing. If GitHub's webhook delivery log fills with 400 responses and Atlantis logs *payload signature check failed*, the webhook secret on the Git host no longer matches the one on the server. Regenerate it and set it in both places, because GitHub never shows you a saved webhook secret again. If comments arrive but plans die on AccessDenied: … is not authorized to perform: sts:AssumeRole, the base role cannot assume the project role. Check that the target role's trust policy names the base role's ARN (Amazon Resource Name, the unique identifier AWS gives every resource). And if plans run fine while Atlantis says nothing on the pull request, the VCS token has expired or lost its scope. App installation tokens renew themselves. A PAT stops working on whatever expiry date its creator picked and then never thought about again.

All of this assumes the server itself is still yours. Short-lived roles and per-run fetches shrink what an attacker walks away with, but a shell on the Atlantis pod still sees every secret *while* a run is happening. That is why the next lesson turns to hardening the server itself: its network exposure, its web UI, its filesystem, and its runtime.

A values.yaml with a GitHub token typed straight into it is exactly as bad as the checked-in .env file from the secrets course. Assume every Helm chart you write ends up public. Inject secrets at deploy time from a sealed source or an external secrets operator. And if a pull-request diff ever printed one, rotate it that day.

Try this

Walk your own Atlantis install and list all five secret classes: cloud credentials, the VCS token, the webhook secret, any extras your workflows pull in, and the credentials for your state backend. Then confirm none of them are baked into image layers or a values file you committed.

kubectl -n atlantis get deploy atlantis -o yaml | grep -iE "name:.*(TOKEN|SECRET|KEY)" | head
# prefer IRSA / workload identity annotations over static keys
kubectl -n atlantis get sa atlantis -o yaml | grep -i role-arn || true

envFrom: secretRef: atlantis-webhook
# serviceAccount annotation: eks.amazonaws.com/role-arn: arn:aws:iam::...:role/atlantis
# no AKIA... in ConfigMap; no token in Dockerfile ENV

Takeaway

Atlantis gathers everyone's keys onto one pegboard, so hold as few as you can: short-lived cloud roles instead of access keys, webhook and VCS secrets kept out of Git, and nothing credential-shaped baked into the image.

Next up for you: move any static cloud keys over to IRSA or your platform's workload identity, rotate the VCS token, and check who can actually read the Kubernetes Secret objects Atlantis mounts.

Related