CoursesAtlantisLocking & the apply gate

Locking & the apply gate

No colliding applies; review first.

Advanced10 min · lesson 4 of 12

A workshop keeps one set of master blueprints. Anyone can photocopy them and sketch a proposal, but only one engineer at a time signs the originals out to the drafting table, and the sign-out sheet by the door records exactly who has them and why. Atlantis locking is that sign-out sheet. Terraform state (the file that records what infrastructure actually exists) is the master blueprint. If two pull requests (PRs) apply against the same state at the same time, one of those plans went stale the moment the other applied, and you get drift, clobbered resources, or a picture of reality that matches nothing. So Atlantis hands out one key per project, and the key stays with the PR that took it.

What a lock actually is

A *project* in Atlantis is the unit that everything hangs off: a repository, one directory inside it, and a Terraform *workspace* (a named copy of state, called default unless you deliberately make more). When a PR runs plan for a project, Atlantis takes a lock on that exact triple: acme-corp/infrastructure + prod/network + default. From that moment, any other PR that tries to plan *or* apply the same triple is refused, with a comment naming the PR that holds the lock. The holder can re-plan as often as it likes and the lock stays put. A different directory, or the same directory in a different workspace, is a different project and locks on its own.

A lock releases in exactly three ways. The holding PR is applied and merged (or closed without ever applying). Someone comments atlantis unlock on the holding PR. Or an operator deletes the lock in the Atlantis web UI (user interface). Now notice what this lock is *not*. It is not Terraform's backend state lock, the DynamoDB item or S3 lock file that stops two terraform processes writing state at the same instant. Atlantis locking sits one floor above that: it serializes *pull requests*, so a plan you reviewed on Monday cannot be quietly invalidated by a different PR applying on Tuesday morning. Run both layers. They fail in different ways.

Will atlantis apply actually run?
PR comments atlantis apply
Two independent gates decide the outcome
Another PR holds the project lock
Blocked
Refused with a comment naming the holding PR; no plan or apply runs until that lock is released
Lock is free, but apply_requirements unmet
Refused by the apply gate
Missing approved, mergeable, or undiverged. These are pinned server-side in repos.yaml so a PR author cannot weaken them
Lock is free and all requirements pass
Apply runs
Terraform applies; merging (or closing) the PR then releases the lock so the next PR can re-plan
The two gates are independent. Bob's apply was refused twice over: locked AND unapproved. Both have to clear before an apply proceeds.
PR #142 — Atlantis comment
Ran Plan for dir: `prod/network` workspace: `default`
**Plan Failed**: This project is currently locked by an unapplied plan
from pull #141. To continue, delete the lock from #141 or apply that
plan and merge the pull request.
Once the lock is released, comment `atlantis plan` here to re-plan.

Where locks live

A lock is a small record: project key, PR number, user, timestamp. It lives in a database on the Atlantis server. By default that database is BoltDB, a single embedded file called atlantis.db inside --data-dir (which defaults to ~/.atlantis). BoltDB is an embedded key/value store that only one process can hold open, which is one reason the standard Atlantis deployment is exactly one replica. If the disk vanishes, the locks vanish with it, and so does Atlantis's memory of every pending plan. --locking-db-type=redis moves lock storage into Redis (a separate in-memory database that runs outside Atlantis) so lock state outlives the server, but it does not make Atlantis horizontally scalable: plan files and cloned working directories still sit on local disk. Either way, run one replica on storage that persists.

server — lock storage
# Default: locks in an embedded BoltDB file under --data-dir
atlantis server \
--atlantis-url="https://atlantis.acme.dev" \
--repo-allowlist="github.com/acme-corp/*" \
--data-dir=/atlantis-data
# lock db lives at /atlantis-data/atlantis.db
# Alternative: externalize lock state to Redis
atlantis server \
--locking-db-type=redis \
--redis-host=redis.infra.svc.cluster.local \
--redis-port=6379 \
--redis-password="$REDIS_PASSWORD" \
--redis-db=0
# Sanity-check the BoltDB file (single file = single replica)
ls -lh /atlantis-data/atlantis.db
# -rw------- 1 atlantis atlantis 512K Jul 13 09:41 /atlantis-data/atlantis.db

The apply gate

Locking decides the order in which changes happen. The apply gate decides whether a change is allowed at all. apply_requirements is the list of conditions a PR has to meet before atlantis apply will do anything: approved (an approving review, judged by the rules in your VCS, the version control system such as GitHub or GitLab), mergeable (required status checks green, no conflicts), and undiverged (the branch is not behind its base, so the plan you reviewed is the plan that merges). Set these in the *server-side* repo config, a repos.yaml you ship with the server via --repo-config. Do not set them in the repository's own atlantis.yaml. The reason is the trust boundary that runs through all of Atlantis: anyone who can open a PR can edit atlantis.yaml, so any control that repo-side config could weaken has to be pinned server-side, with allowed_overrides left empty.

Two flags finish the gate. --gh-team-allowlist="platform-team:plan, platform-leads:apply" says which GitHub teams may run which commands, which turns apply rights into a question of membership rather than a convention people mostly follow. And --disable-apply-all rejects a bare atlantis apply, so every apply has to name its target (-d prod/network -w default). Nobody applies six projects at once because their PR happened to touch six directories.

repos.yaml (server-side)
# Trusted config, deployed with the server — PR authors cannot edit it
repos:
- id: github.com/acme-corp/infrastructure
apply_requirements: [approved, mergeable, undiverged]
allowed_overrides: [] # repo-side atlantis.yaml may NOT weaken the gate

A collision, end to end

Here is the whole dance, with two PRs touching prod/network. Alice opens PR #141. Autoplan runs, the plan posts, the lock is taken. An hour later Bob opens PR #142 against the same directory, and his plan is refused with the lock comment above. He tries atlantis apply anyway and is refused twice over: the project is locked *and* his PR has no approval. Alice's reviewer approves #141, she applies through the gate, and the merge releases the lock. Bob then re-plans against the *new* state, which is the entire point. His plan now includes Alice's changes instead of quietly pretending they never happened.

the collision, as PR comments
## PR #142 — bob tries to apply while locked and unapproved
> bob: atlantis apply -d prod/network -w default
**Apply Failed**: Pull request must be approved according to the
project's approval rules before running apply.
## PR #141 — approved and mergeable; alice applies through the gate
> alice: atlantis apply -d prod/network -w default
Ran Apply for dir: `prod/network` workspace: `default`
Apply complete! Resources: 3 added, 1 changed, 0 destroyed.
## PR #141 merges -> lock released. Back on PR #142:
> bob: atlantis plan
Ran Plan for dir: `prod/network` workspace: `default`
Plan: 2 to add, 0 to change, 0 to destroy.

Deploying so locks survive

On Kubernetes, the official Helm chart (Helm is the package manager for Kubernetes) bakes in the constraints above: a StatefulSet with one replica, and a PersistentVolumeClaim (a request for disk that outlives the pod) mounted at the data directory, so BoltDB locks and pending plan files survive pod restarts and node reschedules. Two values deserve a second look. github.secret is the webhook HMAC secret (HMAC means hash-based message authentication code, a signature computed with a shared key). GitHub signs every delivery with it, in the X-Hub-Signature-256 header, and Atlantis throws away payloads that do not verify. Leave it out and anyone who can reach the webhook endpoint can forge pull-request events. And because the web UI can delete locks and discard plans, put basic authentication in front of it. Every server flag has a matching ATLANTIS_* environment variable, which is how you feed settings through the chart.

Helm deploy
helm repo add runatlantis https://runatlantis.github.io/helm-charts
helm repo update
cat > values.yaml <<'EOF'
orgAllowlist: github.com/acme-corp/*
github:
user: atlantis-bot
token: ghp_xxxx # prod: inject via environmentSecrets, not plaintext
secret: s3cr3t-hmac # webhook secret — verifies X-Hub-Signature-256
volumeClaim:
enabled: true
dataStorage: 8Gi # BoltDB locks + plan files survive restarts
environment:
ATLANTIS_WEB_BASIC_AUTH: "true"
ATLANTIS_WEB_USERNAME: admin # ATLANTIS_WEB_PASSWORD from a secret in prod
EOF
helm install atlantis runatlantis/atlantis -n atlantis --create-namespace -f values.yaml
kubectl get statefulset,pvc -n atlantis
# NAME READY AGE
# statefulset.apps/atlantis 1/1 40s
# NAME STATUS CAPACITY
# persistentvolumeclaim/atlantis-data-atlantis-0 Bound 8Gi

Stuck locks and how to clear them

Sooner or later a lock lingers. A PR was closed without applying, a run died halfway through an apply, or someone went on holiday with an unapplied plan sitting there. The symptom never changes: every other PR on that project fails to plan, pointing at a PR nobody has touched in days. You have two remedies. Comment atlantis unlock on the holding PR, which discards that PR's plans and releases every lock it holds. Or open the Atlantis UI, find the lock on the home page, and click *Discard Plan and Unlock*. If clearing locks should be an operator job only, drop unlock from --allow-commands and keep the UI behind authentication. Then the only people who can clear a lock are the people who can log in.

clearing a stuck lock
# Option 1 — comment on the PR that holds the lock:
atlantis unlock
# Atlantis replies:
# All Atlantis locks for this PR have been unlocked and plans discarded
# Option 2 — web UI: https://atlantis.acme.dev -> click the lock
# -> "Discard Plan and Unlock"
# Option 3 — make unlocking operator-only: drop `unlock` from the
# comment commands so only the authenticated UI can clear locks
# (default: version,plan,apply,unlock,approve_policies,cancel)
atlantis server --allow-commands="version,plan,apply,approve_policies,cancel"
Clear locks deliberately: a force-unlock recreates the exact race locking prevents
Before you clear any lock, open the holding PR and confirm that no apply is genuinely still in flight. Force-unlocking while a run is writing state hands you the concurrent applies this whole mechanism exists to stop. Treat the unlock paths as privileged, too. Anyone who can comment on a PR can issue atlantis unlock, and anyone who can reach an unauthenticated web UI can delete locks without leaving a name. So restrict comment commands with --allow-commands, put the UI behind --web-basic-auth or your SSO (single sign-on) proxy, and write down who cleared the lock and why before the next PR plans.

All of this turned on one word: *project*. How far a lock reaches, who gets blocked by it, and how often two PRs collide are decided entirely by how you carve the repository into projects. One giant root module means every PR queues behind every other. Well-separated directories and workspaces lock on their own and rarely bump into each other. Drawing those boundaries on purpose, with explicit directories, workspaces and autoplan rules in atlantis.yaml, is where we go next.

Locks feel like paperwork right up until the morning two hotfixes collide on the edge VPC (virtual private cloud) module. One apply wins. The other plan was fiction. State locking in the Terraform backend protects the instant of the apply. Atlantis locking protects the human review window before that instant. You want both.

Abandoned PR locks are real, boring toil. Set the expectation out loud: merge or close within a sprint, or unlock and re-plan. A lock held for weeks nearly always means a half-finished change that will startle whoever applies next.

Try this

Open two PRs against the same Atlantis project in a lab repo. Plan on the first one, then watch the second fail on the lock. Clear the lock only when you mean to.

terminal
# PR A
atlantis plan -d prod/network
# PR B (same project)
atlantis plan -d prod/network
atlantis unlock -d prod/network # only after checking who holds it
output
PR A: plan success — lock acquired
PR B: This project is currently locked by an unapplied plan from pull #A
# unlock only when #A is abandoned or applied

Takeaway

A lock is a sign-out sheet for one project: repository plus directory plus workspace. It stops two applies from racing the same state. Clear it with your eyes open.

Next: write down who may unlock, ask for a link to the holding PR in every unlock comment, and never clear a lock to unblock yourself without reading the other plan first.

Quick check
01Your team already runs an S3 backend with DynamoDB state locking, and a colleague argues that the Atlantis project lock adds nothing on top of it. What is the flaw in that argument?
Incorrect — An Atlantis lock is its own small record of project key, pull request number, user and timestamp. It is not a copy of anything the backend wrote, and the two are taken by different things at different moments.
Correct — That is the layering. The backend lock covers the write itself, the Atlantis lock covers the human review window between plan and apply, and you run both because they fail in different ways.
Incorrect — Repository count has nothing to do with it. Two pull requests in one repository touching the same directory and workspace are exactly the collision a project lock exists to stop.
Incorrect — Drop the backend lock and nothing serialises the write itself, so two terraform processes can touch the same state file at the same instant. Neither layer substitutes for the other.
02Pull request #141 has held the lock on prod/network for nine days and its author is on holiday with an unapplied plan sitting there. Which action actually frees the project?
Incorrect — A new commit on the holding branch just triggers a re-plan, and the lock stays exactly where it is. The holder can re-plan as often as it likes without giving anything up.
Incorrect — Atlantis locks carry no timeout. The record sits in BoltDB or Redis until something clears it on purpose, which is why an abandoned pull request can block a project for weeks.
Correct — That is one of the three release paths. The other two are the holder applying and merging, or being closed without applying, and an operator clicking Discard Plan and Unlock in the web UI.
Incorrect — A competing pull request gets refused with a comment naming the holder. It is never handed the lock, no matter how long the holder has been idle.
03Pull request #141 has planned prod/network and holds its lock. Bob opens #142 against the same directory, gets no review on it, and comments atlantis apply -d prod/network -w default. What happens?
Incorrect — Nothing is queued. Bob is refused on the spot, and once #141 merges he has to comment atlantis plan himself so the plan is rebuilt against the state Alice just changed.
Incorrect — Those flags only name the target of the run. Naming a single project does not grant one pull request access to a project another pull request is holding.
Incorrect — A lock stays with the pull request that took it until that pull request applies and merges, closes, or is unlocked. Being the newer plan earns #142 nothing.
Correct — The project lock and apply_requirements are independent checks, and Bob's comment fails each of them on its own.

Related