Runners & executors

What actually runs your jobs.

You push a commit and the pipeline goes nowhere. The build job sits at pending, and the tooltip reads "This job is stuck because no runners are online". Your .gitlab-ci.yml is fine. GitLab works like the order rail in a busy kitchen: it takes the tickets and hangs them up, but it never cooks. A runner is the cook. It is a separate process, usually on a separate machine, that reaches up, takes a ticket, does the work, and shouts the results back. No cook, no dinner. Until you register at least one runner and it matches the job, that build waits forever. This lesson stands a runner up from scratch, reads its config file, compares the three executors, and watches a tagged job land on it.

Two words carry the lesson. A runner is a small program written in Go, the gitlab-runner binary (the compiled agent you install on a machine), and all day it asks GitLab "anything for me?" over and over. That repeated asking has a name: long-polling. An executor is the method that runner uses to actually run your job's script. Straight on the machine's own command line. Inside a throwaway Docker container, which is a sealed disposable box holding its own filesystem and tools. Or as a Kubernetes pod, the smallest unit a cluster schedules, one or more containers that live and die together. The runner is the hand that takes the ticket. The executor decides which kitchen it cooks in, and that is the security-critical call, because it sets how sealed off and how disposable each job's environment is. Other executors exist (ssh, docker-autoscaler, virtualbox), but docker, shell and kubernetes cover almost every real deployment.

Create and register a runner

Modern GitLab (version 16.0 and up, and the only supported path in 17.x) breaks this into two moves. First you create the runner inside GitLab, either in the web interface or through the API (application programming interface, the machine-readable front door to GitLab). Back comes a runner authentication token beginning with glrt-. Creation is also where you set the runner's tags, whether it accepts jobs that carry no tags, and whether it is locked to one project. Second, you register a real machine against that token with gitlab-runner register, which writes the token into config.toml so the agent can prove who it is on every poll. If an older tutorial tells you to paste a "registration token", it is out of date. Those are deprecated and switched off for new projects, so use the API call below. You need a personal access token carrying the create_runner scope, and you should guard it like any admin password.

curl --request POST "https://gitlab.com/api/v4/user/runners" \
  --header "PRIVATE-TOKEN: $GITLAB_PAT" \
  --data "runner_type=project_type" \
  --data "project_id=42" \
  --data "description=docker-linux-01" \
  --data "tag_list=docker,linux" \
  --data "run_untagged=false" \
  --data "locked=true"

{
  "id": 50123,
  "token": "glrt-t3_yXm2K9s4qN8pLwZ1Rb7",
  "token_expires_at": null
}

That glrt- token stands for the runner, not for you. None of your account's power rides along with it, so it is safe to hand to the machine that will do the work. Register it over there. The register command ties this host to that token and picks the executor:

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com" \
  --token "glrt-t3_yXm2K9s4qN8pLwZ1Rb7" \
  --executor "docker" \
  --docker-image "alpine:3.20"

Runtime platform    arch=amd64 os=linux pid=8123 revision=f0a95a76 version=17.11.0
Verifying runner... is valid                        runner=t3_yXm2K9
Registering runner... succeeded                     runner=t3_yXm2K9
Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

The machine is bound now, and its identity lives in config.toml, the single file that controls everything the agent does. On Linux it lands at /etc/gitlab-runner/config.toml by default. Open it. This is where you read and tune the executor, how many jobs run at once, and the shape of the Docker sandbox each job gets.

concurrent = 4
check_interval = 3
 
[[runners]]
  name = "docker-linux-01"
  url = "https://gitlab.com"
  token = "glrt-t3_yXm2K9s4qN8pLwZ1Rb7"
  executor = "docker"
  [runners.docker]
    image = "alpine:3.20"
    privileged = false          # never true on a runner untrusted code can reach
    pull_policy = ["always"]    # re-pull; defeats a stale/tampered local image
    volumes = ["/cache"]        # do NOT add /var/run/docker.sock here

A handful of keys carry most of the weight. concurrent is the ceiling on how many jobs the whole agent runs at the same time, counted across every runner entry it holds, and each [[runners]] block is one registered runner drawing from that budget. check_interval is how often the agent asks GitLab for work, in seconds. Inside [runners.docker], pull_policy decides whether the image is fetched fresh each time. always is the safe answer, because a local copy can go stale or be quietly swapped for a tampered one. Then there is volumes, the list of host folders mounted into every job container. That one line is where isolation is won or lost. Mounting /cache is harmless. Adding /var/run/docker.sock would hand every job the steering wheel of the host's Docker daemon (the background service that builds and runs containers), which is the same as handing it root, the all-powerful administrator account on that machine.

From here the agent runs as a background service, gitlab-runner run, normally kept alive by systemd (the Linux service manager). It keeps asking GitLab for work. When a job it matches appears, it passes the script to the executor, streams the log you watch in the browser, and reports the exit code back. You will rarely type run by hand, but that loop is the thing that moves a job out of pending.

docker vs shell vs kubernetes

The three common executors trade isolation for speed and simplicity. shell runs your script straight on the host as the gitlab-runner user. Fastest, least fuss. But every job shares one filesystem, one set of installed tools, and whatever the last job left lying around, so a single poisoned build can plant a backdoor for the next one and read anything on the box. docker starts a fresh container per job from the image you name, then destroys it. Every job gets a clean room it cannot carry out with it, which is why docker is the sane default. Behind the scenes it also runs a small helper container that handles the git clone, the cache and the artifacts, and drops your script into the build container on its own isolated network bridge. kubernetes does the same trick, except each job becomes a pod scheduled somewhere across a cluster, so the fleet grows with demand and shrinks to zero when the queue empties, with a per-pod security context (non-root user, dropped Linux capabilities, read-only root filesystem) applied to every job. Isolation climbs as you move from shell to docker to kubernetes. So does the amount of machinery you have to operate.

Watch a tagged job land on the runner

So how does one job reach this runner and not some other one? Tags. They work like the skill labels on a job board. A runner advertises what it carries (we set docker,linux at creation), and a job asks for what it needs with tags:. GitLab hands a job to a runner only when the runner's tags cover every tag the job asked for, and because we created this runner with run_untagged=false, it walks past any job that lists no tags at all. Keep tags meaningful: an operating system, a processor architecture, a capability like docker or gpu. One-off names drift and rot. That matching is also an isolation boundary. Give the runners holding production credentials a tag of their own and require it on deploy jobs. An untrusted merge request job cannot edit protected settings, so it can never name its way onto them.

build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]           # kaniko's entrypoint isn't a shell — clear it so the runner can inject sh
  tags: [docker, linux]        # only a runner carrying BOTH tags will take this job
  script:
    - /kaniko/executor --context "$CI_PROJECT_DIR" --dockerfile Dockerfile --no-push

Running with gitlab-runner 17.11.0 (f0a95a76)
  on docker-linux-01 t3_yXm2K9, system ID: s_9c1e4b2a7d10
  feature flags: FF_USE_FASTZIP:true
Preparing the "docker" executor
Using Docker executor with image gcr.io/kaniko-project/executor:v1.23.2-debug ...
Pulling docker image gcr.io/kaniko-project/executor:v1.23.2-debug ...
Preparing environment
Getting source from Git repository
Executing "step_script" stage of the job script
$ /kaniko/executor --context "$CI_PROJECT_DIR" --dockerfile Dockerfile --no-push
INFO[0003] Built cross stage deps: map[]
INFO[0005] Skipping push, --no-push flag set
Job succeeded

In production the trade-off turns concrete. shell gets tempting the moment a job needs the host's graphics cards, a Windows toolchain, or Docker itself. Fine, but that runner should be single tenant, pointed at trusted refs (branches and tags) only, and never sitting in a shared pool. docker costs you an image pull and a container start per job, a few seconds that a warm image and caching mostly hide, and it buys real isolation. kubernetes costs scheduling latency plus a cluster to look after, and repays you in elasticity and that per-pod security context. Default to docker, reach for kubernetes when the queue outgrows one box, and treat shell as a quarantined special case rather than a convenience. Your runner is picking jobs up now, so the next question is how those jobs hand files to each other: artifacts and cache.

Try this

Work through “Watch a tagged job land on the runner” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: config.toml holds a live credential. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related