Runner isolation & protected runners

Keep feature branches off production runners.

Someone you have never met hands you a USB stick and asks you to run the script on it. You would not plug it into the laptop that holds your bank logins. You would use a spare machine you can wipe. A merge request from a fork is that USB stick. A contributor you have never met opens a merge request (an MR, their request to fold their code into your project) from their own copy of your repository, GitLab starts a pipeline to test the change, and one of the jobs runs a script that contributor wrote. If that job lands on the same runner that holds your production deploy token, the script can print $PROD_DEPLOY_TOKEN, POST it to a webhook of their choosing, or break out of the container and take over the host if the runner is in privileged mode. This is not a thought experiment. Untrusted code landing on a trusted runner is the most common way CI systems (continuous integration, the machines that build and test your code) get compromised. Runner isolation exists for one reason: the runner that can reach production must never execute code nobody reviewed.

The runner, the executor, and which way trust flows

A GitLab Runner is a contractor with a key to your building. It is an agent process sitting on a machine you control, asking GitLab whether there is work, then executing each job's script on that machine. The executor is the part that decides where the script actually runs: shell runs it straight on the runner's host, docker runs it inside a fresh container, kubernetes runs it in a throwaway pod. Whatever the script contains, it runs with the runner's privileges, the runner's view of the filesystem, the runner's environment, and the runner's mounted volumes. Every secret the runner can read, the job can read. Every socket the runner mounts, the job can reach. Every Linux capability the runner holds, the job inherits. The key opens whatever the key opens, no matter who wrote the work order. Trust flows down from the runner into the code, which is backwards from what you want when the code is the untrusted part.

Fork merge requests are the sharp edge of all this. When a contributor opens an MR from their fork, GitLab builds the pipeline from the .gitlab-ci.yml (the YAML file, a plain-text config format, that defines your pipeline) and the code sitting in the fork. The contributor writes both. By default that pipeline runs inside the fork's own world: the fork's runners, the fork's CI/CD variables (continuous integration and delivery settings, the values GitLab injects into a job as environment variables), and a CI_JOB_TOKEN scoped to the fork. Your parent project's secrets are not handed over. The trouble starts when a maintainer chooses to run the fork's pipeline in the parent project instead. GitLab allows that through the ci_allow_fork_pipelines_to_run_in_parent_project setting, behind a warning that fork MRs "can contain malicious code that tries to steal secrets in the parent project". Click through it and the job now runs with your runners and your variables while the script stays attacker-controlled. Two things make that combination ugly. If the job lands on a runner with privileged = true or /var/run/docker.sock mounted, the contributor has root on your host. And the job's tags and rules are read from the attacker's YAML, which is the part teams miss, so nothing written in the pipeline file can keep that job away from a sensitive runner. Where a job lands has to be decided by the server, never requested by the pipeline.

Harden how jobs run: config.toml

Start with the runner's own settings file at /etc/gitlab-runner/config.toml. Setting privileged = true is handing a visitor the master key and the alarm code: the container gets the full set of Linux capabilities plus access to the host's devices, which adds up to root on the machine. It exists so people can run Docker-in-Docker (DinD, building container images from inside a container), and it is the first flag an attacker checks for. Mounting /var/run/docker.sock is equally fatal, because any job that can talk to that socket can drive the host's Docker daemon directly and start its own privileged container. Turn both off. To build images without privilege, use Kaniko or rootless BuildKit in place of privileged DinD. Prefer the docker or kubernetes executor so each job gets a fresh, disposable environment and nothing survives between builds. Avoid shell, where jobs share one host and a single poisoned build can leave a backdoor waiting for the next one. Drop the capabilities you do not need while you are in the file.

--- config.toml   (before — insecure)
+++ config.toml   (after — hardened)
 [[runners]]
   name = "prod-deploy"
   url = "https://gitlab.com/"
   token = "glrt-xxxxxxxxxxxxxxxxxxxx"
   executor = "docker"
   [runners.docker]
     image = "alpine:3.20"
-    privileged = true
-    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
+    privileged = false
+    services_privileged = false
+    volumes = ["/cache"]
+    cap_drop = ["ALL"]

Restart the runner with gitlab-runner restart, then prove the change from inside a real job instead of trusting what the file says. The quickest check is to confirm the Docker socket is genuinely gone:

$ docker info
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: exit code 1

Protected runners tie secrets to reviewed code

A protected runner is a bouncer with a guest list. It does not care what your jacket says, it cares whether your name is on the list. Every runner has an access_level setting that decides which refs it will accept work from. Set it to ref_protected and the runner only picks up jobs from protected branches and protected tags, refs that have already been through review and merge control. Pair that with protected CI/CD variables, which GitLab injects only on protected refs, and your fleet splits cleanly in two: unprotected MR pipelines run on ordinary isolated runners holding no production secrets, while the runner carrying your deploy token runs reviewed code from main and nothing else. The property that makes this hold: access_level is enforced by GitLab's job scheduler on the server, not by the pipeline YAML. A contributor can write tags: [prod] in their fork's .gitlab-ci.yml as many times as they like and the protected runner still turns the job away, because the source ref is not protected. Flip the flag through the Runners API (application programming interface):

curl --request PUT \
  --header "PRIVATE-TOKEN: $ADMIN_TOKEN" \
  "https://gitlab.com/api/v4/runners/48213?access_level=ref_protected"

{
  "id": 48213,
  "description": "prod-deploy",
  "active": true,
  "paused": false,
  "runner_type": "project_type",
  "access_level": "ref_protected",
  "tag_list": ["prod"],
  "run_untagged": false
}

Tags handle the positive side of routing, the part ref_protected does not do. Register the production runner with a prod tag and run_untagged = false, then put tags: [prod] on the deploy job and nowhere else. Tags decide which runner a job would like; the protected flag decides which refs a runner is willing to serve. You want both, because tags keep ordinary jobs off the deploy runner day to day, and ref_protected keeps untrusted refs off it on the day somebody forges the tag. Then gate the deploy job itself on CI_COMMIT_REF_PROTECTED, a value GitLab fills in for you on every pipeline, so the job is never even created on an unprotected ref:

stages: [build, deploy]
 
build:
  stage: build
  image: alpine:3.20
  script:
    - ./build.sh          # any shared, isolated runner — no secrets here
 
deploy_prod:
  stage: deploy
  image: alpine:3.20
  tags: [prod]            # prefer the tagged production runner
  rules:
    - if: '$CI_COMMIT_REF_PROTECTED == "true"'   # only on protected refs
  script:
    - echo "token len: ${#PROD_DEPLOY_TOKEN}"
    - ./deploy.sh

# Fork MR whose YAML copies `tags: [prod]` and deletes the rule:
This job is stuck because you don't have any active runners online
or available with any of these tags assigned to them: prod
 
# Same job on protected branch `main`:
Running with gitlab-runner 17.5.0 on prod-deploy 48213 (ref_protected)
$ echo "token len: ${#PROD_DEPLOY_TOKEN}"
token len: 40
Job succeeded

At scale: who registers the runner

Runners come in three scopes. Instance runners are shared across the whole GitLab instance, group runners are shared inside one group, and project runners belong to a single project. On gitlab.com the shared instance runners are already single-use, isolated virtual machines, so the runners you register yourself are where your real exposure sits. Keep the production deploy runner as a dedicated project runner: attached only to the project that deploys, marked ref_protected, and tagged so nothing drifts onto it by accident. For raw build capacity, the autoscaling executors (docker-autoscaler, or the Kubernetes executor) hand every job a fresh instance and destroy it afterwards, which buys you capacity and isolation in one move. The rule that survives a growing fleet: the runners that execute unreviewed code and the runners that hold credentials are two different sets of machines, and no runner belongs to both.

Try this

Work through “At scale: who registers the runner” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a tag is a label, not a lock. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related