Environments, deployments and rollback

Staging on every push, production by a person, and a rollback.

Advanced14 min · lesson 17 of 17
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (20 files, 7 KB): environments.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/ci-gitlab/environments.tar.gz && tar -xzf environments.tar.gz, which creates ~/lab/environments/. SHA-256: 994e2cd49f055a82eecbb9181e01608755c705a47e6b07c95d545b7c399007e2

At 3 a.m. production is broken, and the on-call engineer needs two answers fast: what is running in production right now, and how to put back what ran before. If deploys are scripts that print "done", neither answer is in GitLab. If deploys are jobs with an environment, GitLab records every deployment, and a rollback is re-running an old one.

A target to deploy to

The lab's "production" is a tiny HTTP service that remembers which version each environment runs. A deploy is a PUT with a token, and GET /staging or GET /production shows what's deployed:

target/server.py
"""A pretend deployment target: remembers which version is deployed in each environment.
PUT /<env> with the deploy token and a version in the body; GET /<env> returns the version."""
import os
from http.server import BaseHTTPRequestHandler, HTTPServer
TOKEN = os.environ["DEPLOY_TOKEN"]
deployed = {}
class Target(BaseHTTPRequestHandler):
def do_GET(self):
self._send(200, deployed.get(self.path.strip("/"), "nothing deployed"))
def do_PUT(self):
if self.headers.get("Authorization") != f"Bearer {TOKEN}":
return self._send(401, "bad token")
body = self.rfile.read(int(self.headers.get("Content-Length", 0))).decode()
deployed[self.path.strip("/")] = body
self._send(200, f"{self.path.strip('/')} now runs {body}")
def _send(self, code, text):
self.send_response(code)
self.end_headers()
self.wfile.write((text + "\n").encode())
def log_message(self, *args):
pass
HTTPServer(("0.0.0.0", 8000), Target).serve_forever()
app/.gitlab-ci.yml
# Staging deploys on every push to main. Production waits for a person, and only one
# production deployment runs at a time.
stages: [build, staging, production]
.deploy:
image: secopslog-py:3.15
script:
- python deploy.py
build:
stage: build
image: secopslog-py:3.15
script:
- python -m py_compile calc.py && echo "built $CI_COMMIT_SHORT_SHA"
deploy-staging:
stage: staging
extends: .deploy
environment:
name: staging
url: http://127.0.0.1:8000/staging
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
deploy-production:
stage: production
extends: .deploy
environment:
name: production
url: http://127.0.0.1:8000/production
resource_group: production
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: manual

environment turns a job into a deployment that GitLab tracks. deploy-staging runs on every push to main. deploy-production is when: manual, so it waits for someone to start it, and resource_group: production makes sure only one production deployment runs at a time, even if two pipelines reach it together. The deploy token is a protected, masked variable, so only main and other protected refs can deploy.

ubuntu@secopslog-cicd:~/lab/environments · GitLab 19.5.3
$ ./up.sh && echo "DEPLOY_TOKEN=$(openssl rand -hex 16)" >> .env && docker compose -f compose.yaml -f target.yaml --progress quiet up -d target && curl -fs --retry 10 --retry-all-errors --retry-delay 1 http://127.0.0.1:8000/staging
runner lab-runner: online (tags: lab,docker) GitLab 19.5.3 is up on http://127.0.0.1:8929 nothing deployed
$ ./seed.sh app app/ && QUIET=1 ./pipeline.sh app/ >/dev/null; . ./.env && ./gl.sh /projects/root%2Fapp/variables -X POST -d key=DEPLOY_TOKEN -d "value=$DEPLOY_TOKEN" -d masked=true -d protected=true -o /dev/null && ./gl.sh /projects/root%2Fapp/pipeline -X POST -d ref=main -o /dev/null && sleep 3 && ./pipeline.sh app/
pushed app: 0d993bb first commit --- job build (build): success $ python -m py_compile calc.py && echo "built $CI_COMMIT_SHORT_SHA" built 0d993bb5 --- job deploy-staging (staging): success $ python deploy.py staging now runs 0d993bb5 --- job deploy-production (production): manual pipeline #2: manual
$ for e in staging production; do printf "%s: " $e; curl -fsS http://127.0.0.1:8000/$e; done
staging: 0d993bb5 production: nothing deployed

The seed push ran a pipeline before DEPLOY_TOKEN existed, so the step waited for it, created the variable and started a new pipeline through the API. That one ran build and deploy-staging and stopped at deploy-production, which is manual. Staging now runs 0d993bb5, and production has nothing yet. A pipeline that ends at a manual job has the status manual in the API (blocked in the UI): it is waiting for a decision.

Deploy to production

Starting a manual job is the "play" button in the UI. play.sh does the same through the API and waits:

ubuntu@secopslog-cicd:~/lab/environments · GitLab 19.5.3
$ ./play.sh app/ deploy-production && curl -fsS http://127.0.0.1:8000/production
played deploy-production (job 6), now waiting_for_resource deploy-production: success 0d993bb5
$ echo "# v2" >> app/calc.py && ./commit.sh app/ "v2" && ./pipeline.sh app/ | grep -E "^(--- job|staging|pipeline)" && ./play.sh app/ deploy-production && curl -fsS http://127.0.0.1:8000/production
pushed: ef619b8 v2 to main --- job build (build): success --- job deploy-staging (staging): success staging now runs ef619b8d --- job deploy-production (production): manual pipeline #3: manual played deploy-production (job 9), now waiting_for_resource deploy-production: success ef619b8d
$ ./gl.sh '/projects/root%2Fapp/deployments?environment=production&order_by=id&sort=asc' | jq -r '.[] | "deployment #\(.iid) \(.status) sha=\(.sha[0:7]) job=\(.deployable.id)"'
deployment #2 skipped sha=0d993bb job=3 deployment #4 success sha=0d993bb job=6 deployment #6 success sha=ef619b8 job=9

Each played job first showed waiting_for_resource while it took the production resource group, then ran. Production went to 0d993bb5, then to ef619b8d. The deployment list is GitLab's record of production: which commit, from which job. The first entry is skipped: it belongs to the seed push's pipeline, whose staging deploy failed without the token, so its production job never became available.

Who may press play? Add dev as a Developer, push another change, and let dev try to start that pipeline's production job, then root:

ubuntu@secopslog-cicd:~/lab/environments · GitLab 19.5.3
$ ./user.sh dev >/dev/null && ./member.sh app dev 30 >/dev/null && echo '# v3' >> app/calc.py && ./commit.sh app/ v3 >/dev/null && ./pipeline.sh app/ >/dev/null; ./as.sh dev ./gl.sh /projects/root%2Fapp/jobs/$(./gl.sh '/projects/root%2Fapp/jobs?scope=manual' | jq -r '[.[] | select(.name=="deploy-production")][0].id')/play -X POST
curl: (22) The requested URL returned error: 403 {"message":"403 Forbidden"}
$ ./play.sh app/ deploy-production && curl -fsS http://127.0.0.1:8000/production
played deploy-production (job 12), now waiting_for_resource deploy-production: success e9747897

GitLab refused dev with 403 Forbidden: starting a manual job needs permission to merge into its branch, and only Maintainers may merge into the protected main. Root played the same job, and production moved to e9747897, version 3.

Roll back

Version 3 is bad. Roll back by re-running the deployment job of the last good version, the successful deployment before the current one, which deploys that commit again:

ubuntu@secopslog-cicd:~/lab/environments · GitLab 19.5.3
$ jid=$(./gl.sh '/projects/root%2Fapp/deployments?environment=production&status=success&order_by=id&sort=desc' | jq -r '.[1].deployable.id') && new=$(./gl.sh /projects/root%2Fapp/jobs/$jid/retry -X POST | jq -r .id) && echo "retried job $jid as job $new" && for _ in $(seq 60); do st=$(./gl.sh /projects/root%2Fapp/jobs/$new | jq -r .status); case $st in success|failed) break ;; esac; sleep 2; done && echo "job $new: $st" && curl -fsS http://127.0.0.1:8000/production && eid=$(./gl.sh '/projects/root%2Fapp/environments?name=production' | jq -r '.[0].id') && ./gl.sh /projects/root%2Fapp/environments/$eid | jq -r '"environment \(.name): last deployment \(.last_deployment.sha[0:8]) by job \(.last_deployment.deployable.id)"'
retried job 9 as job 13 job 13: success ef619b8d environment production: last deployment ef619b8d by job 13

The deployment before the current one was job 9, version 2, ef619b8d. Retrying it created job 13, which deployed that same commit again. Production answers ef619b8d, and the environment records the rollback as its newest deployment, by job 13. Nothing was rebuilt: the retried job deployed the commit it was created for, which is why a rollback this way is quick.

Watch out
A retried job is the old job, not the old world. It runs the old commit's .gitlab-ci.yml and deploy script against today's infrastructure, with today's variable values, and it needs the old pipeline's artifacts, which may have expired. It doesn't undo database migrations either. The retry worked here because "Prevent outdated deployment jobs" allows retries for rollbacks by default. The same setting stops an old manual job from being played after a newer deployment and putting an old commit back.

Three more settings belong on a real production job. resource_group takes jobs in no particular order by default, so set process_mode: oldest_first if deployments must land in pipeline order. Mark deploy jobs interruptible: false, so auto-cancelling of redundant pipelines can't stop one halfway. And give each environment its own credential. Here one DEPLOY_TOKEN serves staging and production, so the automatic staging job holds a production-capable secret. Scope variables to an environment, or use ID tokens bound to it.

FeatureFree tierUse
Environments and deployment historyYesWhat runs where, deployed by whom, from which commit
Manual jobs (when: manual)YesA person starts production deploys
resource_groupYesOne job at a time per resource group (here, one per environment)
Re-running an old deploymentYesRollback to a known version
Protected environments and deployment approvalsNo, PremiumRestrict who may deploy, require approvers
Watch out
On the Free tier, two things decide who can deploy. Starting a manual job needs permission to merge into its branch, so on a protected main only the people allowed to merge can press play. And the deploy only works where the deploy credential arrives, so keep it in a protected variable or behind an ID token. Protected environments, a paid feature, add a separate list of deployers on top.

Hands-on

Start two production deployments at once. Let the runner take two jobs (docker compose exec -T runner sed -i 's/^concurrent = 1/concurrent = 2/' /etc/gitlab-runner/config.toml), push two changes, each with ./commit.sh and ./pipeline.sh app/, and then play both pipelines' deploy-production jobs within a second of each other, with ./gl.sh /projects/root%2Fapp/jobs/JOB_ID/play -X POST (the IDs are in ./gl.sh '/projects/root%2Fapp/jobs?scope=manual'). Check both jobs' status straight away, and again half a minute later. What does resource_group do? Finish with ./down.sh.

What you should see: the runner config shows concurrent = 2, and both jobs were played within a second. Four seconds later, job 16 (1e915fa9, v4) was running and job 19 (4cc23090, v5) was waiting_for_resource, although the runner had a free slot. After both finished, production runs 4cc23090. Without resource_group the two deploys would have run at the same time and overwritten each other in whatever order they finished. With "Prevent outdated deployment jobs" on, an older job that starts after a newer one has deployed is refused instead.

Remember

Give every deploy job an environment, make production manual and one-at-a-time with resource_group, and practise the rollback before you need it.

Quick check
01What does adding environment: name: production to a deploy job give you?
Incorrect — GitLab records deployments. It does not roll back on its own.
Incorrect — Runners are chosen by tags and protection, not by environment.
Incorrect — That needs protected environments, a paid feature.
Correct — GitLab records every deployment to production. That history is what makes a rollback a re-run instead of an investigation.
02Two pipelines reach deploy-production at the same moment. The job has resource_group: production. What happens?
Correct — Jobs in one resource group never overlap.
Incorrect — That is what happens without a resource group.
Incorrect — It waits, it is not cancelled.
Incorrect — Resource groups queue jobs. They do not fail them.
03On the Free tier, what limits who can start deploy-production on a protected main?
Incorrect — resource_group only orders jobs and checks no roles.
Correct — Starting a manual job needs permission to merge into its branch. The protected deploy credential is the other half: the job works only where it arrives.
Incorrect — Environment names are not restricted on the Free tier.
Incorrect — when: manual only makes the job wait. It checks no role.

Related