Environments, deployments and rollback
Staging on every push, production by a person, and a rollback.
cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/ci-gitlab/environments.tar.gz && tar -xzf environments.tar.gz, which creates ~/lab/environments/. SHA-256: 994e2cd49f055a82eecbb9181e01608755c705a47e6b07c95d545b7c399007e2At 3 a.m. production is broken, and the on-call engineer needs two answers fast: what is running in production right now, and how to put back what ran before. If deploys are scripts that print "done", neither answer is in GitLab. If deploys are jobs with an environment, GitLab records every deployment, and a rollback is re-running an old one.
A target to deploy to
The lab's "production" is a tiny HTTP service that remembers which version each environment runs. A deploy is a PUT with a token, and GET /staging or GET /production shows what's deployed:
"""A pretend deployment target: remembers which version is deployed in each environment.PUT /<env> with the deploy token and a version in the body; GET /<env> returns the version."""import osfrom http.server import BaseHTTPRequestHandler, HTTPServerTOKEN = os.environ["DEPLOY_TOKEN"]deployed = {}class Target(BaseHTTPRequestHandler):def do_GET(self):self._send(200, deployed.get(self.path.strip("/"), "nothing deployed"))def do_PUT(self):if self.headers.get("Authorization") != f"Bearer {TOKEN}":return self._send(401, "bad token")body = self.rfile.read(int(self.headers.get("Content-Length", 0))).decode()deployed[self.path.strip("/")] = bodyself._send(200, f"{self.path.strip('/')} now runs {body}")def _send(self, code, text):self.send_response(code)self.end_headers()self.wfile.write((text + "\n").encode())def log_message(self, *args):passHTTPServer(("0.0.0.0", 8000), Target).serve_forever()
# Staging deploys on every push to main. Production waits for a person, and only one# production deployment runs at a time.stages: [build, staging, production].deploy:image: secopslog-py:3.15script:- python deploy.pybuild:stage: buildimage: secopslog-py:3.15script:- python -m py_compile calc.py && echo "built $CI_COMMIT_SHORT_SHA"deploy-staging:stage: stagingextends: .deployenvironment:name: stagingurl: http://127.0.0.1:8000/stagingrules:- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHdeploy-production:stage: productionextends: .deployenvironment:name: productionurl: http://127.0.0.1:8000/productionresource_group: productionrules:- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHwhen: manual
environment turns a job into a deployment that GitLab tracks. deploy-staging runs on every push to main. deploy-production is when: manual, so it waits for someone to start it, and resource_group: production makes sure only one production deployment runs at a time, even if two pipelines reach it together. The deploy token is a protected, masked variable, so only main and other protected refs can deploy.
The seed push ran a pipeline before DEPLOY_TOKEN existed, so the step waited for it, created the variable and started a new pipeline through the API. That one ran build and deploy-staging and stopped at deploy-production, which is manual. Staging now runs 0d993bb5, and production has nothing yet. A pipeline that ends at a manual job has the status manual in the API (blocked in the UI): it is waiting for a decision.
Deploy to production
Starting a manual job is the "play" button in the UI. play.sh does the same through the API and waits:
Each played job first showed waiting_for_resource while it took the production resource group, then ran. Production went to 0d993bb5, then to ef619b8d. The deployment list is GitLab's record of production: which commit, from which job. The first entry is skipped: it belongs to the seed push's pipeline, whose staging deploy failed without the token, so its production job never became available.
Who may press play? Add dev as a Developer, push another change, and let dev try to start that pipeline's production job, then root:
GitLab refused dev with 403 Forbidden: starting a manual job needs permission to merge into its branch, and only Maintainers may merge into the protected main. Root played the same job, and production moved to e9747897, version 3.
Roll back
Version 3 is bad. Roll back by re-running the deployment job of the last good version, the successful deployment before the current one, which deploys that commit again:
The deployment before the current one was job 9, version 2, ef619b8d. Retrying it created job 13, which deployed that same commit again. Production answers ef619b8d, and the environment records the rollback as its newest deployment, by job 13. Nothing was rebuilt: the retried job deployed the commit it was created for, which is why a rollback this way is quick.
.gitlab-ci.yml and deploy script against today's infrastructure, with today's variable values, and it needs the old pipeline's artifacts, which may have expired. It doesn't undo database migrations either. The retry worked here because "Prevent outdated deployment jobs" allows retries for rollbacks by default. The same setting stops an old manual job from being played after a newer deployment and putting an old commit back.Three more settings belong on a real production job. resource_group takes jobs in no particular order by default, so set process_mode: oldest_first if deployments must land in pipeline order. Mark deploy jobs interruptible: false, so auto-cancelling of redundant pipelines can't stop one halfway. And give each environment its own credential. Here one DEPLOY_TOKEN serves staging and production, so the automatic staging job holds a production-capable secret. Scope variables to an environment, or use ID tokens bound to it.
| Feature | Free tier | Use |
|---|---|---|
| Environments and deployment history | Yes | What runs where, deployed by whom, from which commit |
Manual jobs (when: manual) | Yes | A person starts production deploys |
resource_group | Yes | One job at a time per resource group (here, one per environment) |
| Re-running an old deployment | Yes | Rollback to a known version |
| Protected environments and deployment approvals | No, Premium | Restrict who may deploy, require approvers |
main only the people allowed to merge can press play. And the deploy only works where the deploy credential arrives, so keep it in a protected variable or behind an ID token. Protected environments, a paid feature, add a separate list of deployers on top.Hands-on
Start two production deployments at once. Let the runner take two jobs (docker compose exec -T runner sed -i 's/^concurrent = 1/concurrent = 2/' /etc/gitlab-runner/config.toml), push two changes, each with ./commit.sh and ./pipeline.sh app/, and then play both pipelines' deploy-production jobs within a second of each other, with ./gl.sh /projects/root%2Fapp/jobs/JOB_ID/play -X POST (the IDs are in ./gl.sh '/projects/root%2Fapp/jobs?scope=manual'). Check both jobs' status straight away, and again half a minute later. What does resource_group do? Finish with ./down.sh.
What you should see: the runner config shows concurrent = 2, and both jobs were played within a second. Four seconds later, job 16 (1e915fa9, v4) was running and job 19 (4cc23090, v5) was waiting_for_resource, although the runner had a free slot. After both finished, production runs 4cc23090. Without resource_group the two deploys would have run at the same time and overwritten each other in whatever order they finished. With "Prevent outdated deployment jobs" on, an older job that starts after a newer one has deployed is refused instead.
Remember
Give every deploy job an environment, make production manual and one-at-a-time with resource_group, and practise the rollback before you need it.
environment: name: production to a deploy job give you?deploy-production at the same moment. The job has resource_group: production. What happens?deploy-production on a protected main?resource_group only orders jobs and checks no roles.when: manual only makes the job wait. It checks no role.