Rules, needs and merge request pipelines

Which pipelines run, which jobs join them, and in what order.

Intermediate15 min · lesson 3 of 17
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (18 files, 6 KB): rules.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/ci-gitlab/rules.tar.gz && tar -xzf rules.tar.gz, which creates ~/lab/rules/. SHA-256: d78d8d133c510a735d1e934eba376ee61a506b0ac7409282ab8d083d6421b69f

A developer opens a merge request and sees two pipelines for the same commit, one for the branch and one for the merge request, each running the full test suite. Another team's deploy job runs on every feature branch because nobody told it not to. Both problems come from the same place: a job without rules joins every pipeline its project creates, and once anything asks for merge request pipelines, a branch can get both kinds. It's up to you to say which pipelines exist and which jobs join them.

This lesson covers the three decisions a pipeline makes: whether to create a pipeline at all (workflow:rules), which jobs join it (job rules), and in what order they run (stages and needs).

The pipeline file

app/.gitlab-ci.yml
# One pipeline per change: a merge request pipeline when an MR is open, otherwise a branch pipeline.
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_COMMIT_TAG
default:
image: secopslog-py:3.15
stages: [check, test, deploy]
slow-lint:
stage: check
script:
- sleep 15
- ruff check .
build:
stage: check
script:
- python -m compileall -q .
- echo built
test:
stage: test
needs: [build] # starts as soon as build is done, without waiting for slow-lint
parallel:
matrix:
- SUITE: [unit, smoke]
script:
- echo "suite $SUITE"
- python -m pytest -q
docs:
stage: test
needs: []
rules:
- changes: [docs/**/*]
script:
- echo "docs changed"
deploy:
stage: deploy
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
script:
- echo "deploying $CI_COMMIT_SHORT_SHA"

Rules are checked from top to bottom, and the first one that matches decides. In workflow:rules, a match creates the pipeline, and when: never prevents it. The second rule is the one that stops duplicates: a branch push while a merge request is open for that branch creates no branch pipeline, because the merge request pipeline covers it. If no rule matches, there is no pipeline.

Job rules work the same way, per job. deploy joins only on the default branch. docs joins when files under docs/ changed. test uses needs: [build], so it starts as soon as build finishes, without waiting for the rest of its stage. And parallel:matrix turns one job definition into one job per combination of values.

A push to main

Let the runner take three jobs at once first, as in the runners lesson, so needs has room to make a difference:

ubuntu@secopslog-cicd:~/lab/rules · GitLab 19.5.3
$ ./up.sh
runner lab-runner: online (tags: lab,docker) GitLab 19.5.3 is up on http://127.0.0.1:8929
$ docker compose exec -T runner sed -i 's/^concurrent = 1/concurrent = 3/' /etc/gitlab-runner/config.toml && sleep 4 && docker compose logs --no-log-prefix runner 2>&1 | grep -o 'Configuration loaded.*' | tail -1
Configuration loaded builds=0 max_builds=3
$ ./seed.sh app app/ && QUIET=1 PIPE_WAIT=900 ./pipeline.sh app/
pushed app: fb7467b first commit --- job slow-lint (check): success --- job build (check): success --- job test: [unit] (test): success --- job test: [smoke] (test): success --- job docs (test): success --- job deploy (deploy): success pipeline #1: success
$ ./gl.sh "/projects/root%2Fapp/pipelines?per_page=1" | jq -r '.[0].id' | xargs -I{} ./gl.sh '/projects/root%2Fapp/pipelines/{}/jobs?per_page=50' | jq -r 'sort_by(.started_at) | .[] | "\(.name | .[0:16]) started \(.started_at[11:19]) finished \(.finished_at[11:19])"'
slow-lint started 03:19:26 finished 03:19:44 build started 03:19:26 finished 03:19:29 docs started 03:19:26 finished 03:19:29 test: [unit] started 03:19:32 finished 03:19:35 test: [smoke] started 03:19:33 finished 03:19:35 deploy started 03:19:45 finished 03:19:46

Read the start times. build, slow-lint and docs started in the same second. docs has needs: [], so it didn't wait for the check stage. Both test jobs, one per matrix value, started at 03:19:32 and 03:19:33, three seconds after build finished and while slow-lint was still sleeping. deploy has no needs, so it waited for the whole test stage and also for slow-lint, and started at 03:19:45, a second after slow-lint finished. docs would have run anyway, since this push added docs/usage.md, but note the rule behind it: on the first push of a branch there is nothing to compare with, so changes is always true. To compare a new branch with main instead, write changes: { paths: [docs/**/*], compare_to: refs/heads/main }.

A feature branch

ubuntu@secopslog-cicd:~/lab/rules · GitLab 19.5.3
$ printf "\n\ndef sub(a, b):\n return a - b\n" >> app/calc.py && ./commit.sh app/ "add sub" feature/sub && QUIET=1 PIPE_WAIT=900 ./pipeline.sh app/
pushed: 5485479 add sub to feature/sub --- job slow-lint (check): success --- job build (check): success --- job test: [unit] (test): success --- job test: [smoke] (test): success --- job docs (test): success pipeline #2: success
$ printf "\n\ndef mul(a, b):\n return a * b\n" >> app/calc.py && ./commit.sh app/ "add mul" && QUIET=1 PIPE_WAIT=900 ./pipeline.sh app/
pushed: 9366b0e add mul to feature/sub --- job slow-lint (check): success --- job build (check): success --- job test: [unit] (test): success --- job test: [smoke] (test): success pipeline #3: success

There's no deploy on the branch, as intended. Look at docs, though. The first push to feature/sub ran it although nothing under docs/ changed, and the second push skipped it. For a branch pipeline, changes compares with the previous push to that branch, and a branch's first push has nothing to compare with, so GitLab treats every file as changed. Any rule like this runs on every new branch.

A merge request

ubuntu@secopslog-cicd:~/lab/rules · GitLab 19.5.3
$ ./gl.sh '/projects/root%2Fapp/merge_requests' -X POST -d source_branch=feature/sub -d target_branch=main -d 'title=Add sub and mul' | jq -r '"MR !\(.iid) \(.state): \(.source_branch) -> \(.target_branch)"'; sleep 5; QUIET=1 PIPE_WAIT=900 ./pipeline.sh app/
MR !1 opened: feature/sub -> main --- job slow-lint (check): success --- job build (check): success --- job test: [unit] (test): success --- job test: [smoke] (test): success pipeline #4: success
$ printf "\n\ndef div(a, b):\n return a / b\n" >> app/calc.py && ./commit.sh app/ "add div" && QUIET=1 PIPE_WAIT=900 ./pipeline.sh app/ >/dev/null; ./gl.sh "/projects/root%2Fapp/pipelines?sha=$(git -C app rev-parse HEAD)" | jq -r ".[] | \"pipeline #\\(.id) source=\\(.source) ref=\\(.ref) \\(.status)\""
pushed: 66b32e4 add div to feature/sub pipeline #5 source=merge_request_event ref=refs/merge-requests/1/head success

The merge request pipeline skipped docs, because in merge request pipelines changes compares the branch with the target branch, which is the comparison you actually want. And the push after the merge request opened produced exactly one pipeline, from merge_request_event. Its ref is refs/merge-requests/1/head, not feature/sub, which matters later: protected variables follow protected refs, and this ref is not one. (A project setting can let merge request pipelines receive them when both the source and target branches are protected, in the same project.)

VariableBranch pipelineMerge request pipelineTag pipeline
CI_PIPELINE_SOURCEpushmerge_request_eventpush
CI_COMMIT_BRANCHthe branchnot setnot set
CI_COMMIT_TAGnot setnot setthe tag
changes compares withthe previous pushthe target branchnothing: always true
Watch out
Rules decide what runs. They aren't access control. A rule that limits deploy to the default branch sits in a file every developer can edit on their own branch. Secrets and production access need protected branches and protected variables, from later lessons.

Hands-on

Before you run it, predict which jobs a tag pipeline gets from this file. Then go back to main and push a tag: git -C app checkout main && git -C app pull && git -C app tag v0.1.0 && git -C app push origin v0.1.0, and list the jobs with QUIET=1 ./pipeline.sh app/. Were you right about docs and deploy? Finish with ./down.sh.

What you should see: the tag pipeline ran slow-lint, build, both test jobs and docs, and no deploy. deploy needs $CI_COMMIT_BRANCH, which is empty in a tag pipeline, so the rule didn't match. docs joined because a tag pipeline has no previous push to compare with, so changes is true again. If tags are your releases, give the release jobs a rule on $CI_COMMIT_TAG.

Remember

workflow:rules decides whether a pipeline exists, job rules decide who joins, and needs decides who waits for whom. Check changes rules on a branch's first push, and keep secrets out of reach of anything a rule alone protects.

Quick check
01A project has no workflow:rules. A developer opens a merge request and pushes again. Each push now runs the whole pipeline twice. Why?
Incorrect — GitLab runs what the rules allow. Without workflow rules, both kinds can be created.
Incorrect — Runners do not retry by default, and that would not create a second pipeline.
Correct — Jobs with merge request rules run in both unless workflow rules prevent duplicates. The $CI_OPEN_MERGE_REQUESTS rule with when: never drops the branch pipeline while a merge request is open.
Incorrect — needs orders jobs inside one pipeline. It never creates pipelines.
02A job has rules: - changes: [docs/**/*]. A developer pushes a brand-new branch that only touches src/. Does the job run in the branch pipeline?
Correct — The lab saw it: docs ran on the first push to feature/sub and not on the second.
Incorrect — That is how it works on later pushes. The first push of a branch is the exception.
Incorrect — Branch names are not part of a changes rule.
Incorrect — changes works in both. It just compares with different things.
03test has needs: [build], and slow-lint is in the same stage as build. With enough runner capacity, when does test start?
Incorrect — That is the default without needs. With needs, a job waits only for the jobs it names.
Correct — needs turns the stage order into a graph of real dependencies.
Incorrect — needs means "after". The job waits for build to finish.
Incorrect — needs has nothing to do with manual jobs.

Related