Hardening self-hosted GitLab runners

Isolate jobs, scope CI tokens, and pin executors so one poisoned pipeline can't pivot into your whole GitLab runner fleet or leak another team's secrets.

Aug 13, 2024·Updated ·7 min readAdvanced·By SecOpsLog · documentation-verified

A runner executes whatever a pipeline says, and a pipeline is a file anyone with push access can edit. On a self-hosted runner that means the job script is untrusted code running on a machine that also holds the runner's registration token, cached artifacts from other projects, and often a Docker socket. Hardening is the set of choices that stop one malicious or compromised job from becoming a host takeover or a read of another team's secrets, and most of those choices are three lines in config.toml plus a few project settings.

The advice has moved since the first wave of "use kaniko" articles. kaniko is no longer maintained and GitLab has removed its kaniko guidance; the documented rootless options for building images are now Buildah, BuildKit in rootless mode and Podman. The rest of the model is unchanged: no privileged containers, no socket, one clean workspace per job, tokens that reach only what they must.

Privileged mode and the socket are the same decision

GitLab's own security page puts it plainly: privileged containers have all the root capabilities of the host, and the same page marks the host PID namespace as unsafe. Binding /var/run/docker.sock into jobs is the other way to reach the same place, because a job that can talk to the daemon can start a privileged container with the host root filesystem mounted. Docker-in-Docker is the usual reason both settings get switched on, so removing the reason removes the settings: build images with a tool that does not need a daemon, on a host where the daemon itself is rootless if it has to exist at all.

/etc/gitlab-runner/config.toml
[[runners]]
name = "build-general"
executor = "docker"
[runners.docker]
privileged = false
volumes = ["/cache"] # never "/var/run/docker.sock:/var/run/docker.sock"
cap_drop = ["ALL"]
cap_add = ["CHOWN", "SETUID", "SETGID", "DAC_OVERRIDE"] # what ordinary builds need
pull_policy = ["always"]
allowed_pull_policies = ["always"]
.gitlab-ci.yml (image build without a daemon)
build-image:
image: quay.io/buildah/stable
variables:
STORAGE_DRIVER: vfs
BUILDAH_FORMAT: docker
before_script:
- echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
script:
- buildah build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
- buildah push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
bash — the two greps that find an exposed runner
grep -nE "privileged *= *true|docker\.sock|pid *= *\"host\"" /etc/gitlab-runner/config.toml
14: privileged = true
17: volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
either line means every job that ran here could have been root on the host: rotate the secrets that host could see, then fix the config
gitlab-runner verify && gitlab-runner --version | head -1
Verifying runner... is valid runner=build-general
Version: 18.4.0

CI_JOB_TOKEN reaches only what the allowlist says

CI_JOB_TOKEN is short-lived but it acts with the permissions of the user who triggered the pipeline, against the GitLab API, for the duration of the job. What limits it is the job token allowlist on the target project: a job in project A can use its token against project B only if B has added A to its allowlist. That default has been in place since the allowlist became the standard mechanism, so the work is to audit projects that still have the old "any project" behaviour enabled and to keep allowlists minimal rather than adding the whole group.

Deploy credentials belong in protected variables, which are only injected into pipelines on protected branches and tags, and ideally in masked ones so they never appear in job logs. The combination that leaks secrets is a variable that is neither, in a project where anyone can push a branch: the branch's pipeline runs the attacker's script with the variable in its environment. Pipeline secrets covers the variable-scoping side in more depth.

Fork merge requests run in the fork

A merge request from a fork runs its pipeline in the fork project by default, with the fork's variables and the fork's runners, so external contributions cannot see your protected variables. The setting that changes this, running fork pipelines in the parent project, is the one to be careful with: it makes the parent's runners and non-protected variables available to whatever the fork's .gitlab-ci.yml says, so it should be paired with the requirement that a maintainer reviews the changes before the pipeline runs. Since GitLab 18.1 access to protected variables and protected runners for merge request pipelines is controlled explicitly, and fork pipelines never get them.

After privileged mode, assume the host was compromised
With privileged = true or a mounted socket, any job could read /etc/gitlab-runner/config.toml, the registration token in it, and every variable cached on the host for every project the runner served. Fixing the setting is not enough: rotate the runner token and the CI variables those projects use, review the job history for the period the setting was on, and rebuild the host from a clean image.

Split runners by what they can reach

One fleet for everything means the runner that runs an unknown dependency's postinstall script is the runner that holds the production kubeconfig. Tags and protected runners separate them: general build and test runners register with build tags and hold no deploy credentials; a small pool of runners marked protected in GitLab, on hosts in a locked network segment, register with deploy tags and only ever pick up jobs from protected refs. Autoscaled or ephemeral runners add a third property, a fresh disk per job, so nothing one job leaves behind is readable by the next.

.gitlab-ci.yml (jobs pick their trust zone)
test:
tags: [build]
script: [make test]
deploy-prod:
tags: [deploy] # only the protected runner pool has this tag
rules:
- if: $CI_COMMIT_BRANCH == "main"
environment: production
script: [./deploy.sh]

Runner pools by trust zone

PoolJobsHoldsRunner settings
build (shared)test, lint, image buildregistry push token onlyunprivileged Docker executor, no socket, ephemeral
deploy (protected)deploy-prod, migrationskubeconfig or cloud OIDC roleprotected runner, protected refs only, locked network
shell (avoid)legacy scriptsthe runner user’s whole accounttrusted projects only, per GitLab’s own guidance

With the runners split, scanning jobs can run on the shared pool and signing or deploy jobs on the protected one, and the deploy job can obtain its cloud credentials through OIDC rather than a stored key, which is the same pattern as GitHub Actions to AWS with OIDC with GitLab's id_tokens. None of it requires new tooling; it requires deciding which runners are allowed to hold which secrets and encoding that decision in tags, protection flags and the allowlist.

Go deeper in a courseSecure CI/CD with GitLabRunner isolation, scoped tokens, protected pipelines and scanning gates.View course

Related posts

Quick reference