Rules, needs and merge request pipelines
Which pipelines run, which jobs join them, and in what order.
cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/ci-gitlab/rules.tar.gz && tar -xzf rules.tar.gz, which creates ~/lab/rules/. SHA-256: d78d8d133c510a735d1e934eba376ee61a506b0ac7409282ab8d083d6421b69fA developer opens a merge request and sees two pipelines for the same commit, one for the branch and one for the merge request, each running the full test suite. Another team's deploy job runs on every feature branch because nobody told it not to. Both problems come from the same place: a job without rules joins every pipeline its project creates, and once anything asks for merge request pipelines, a branch can get both kinds. It's up to you to say which pipelines exist and which jobs join them.
This lesson covers the three decisions a pipeline makes: whether to create a pipeline at all (workflow:rules), which jobs join it (job rules), and in what order they run (stages and needs).
The pipeline file
# One pipeline per change: a merge request pipeline when an MR is open, otherwise a branch pipeline.workflow:rules:- if: $CI_PIPELINE_SOURCE == "merge_request_event"- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTSwhen: never- if: $CI_COMMIT_BRANCH- if: $CI_COMMIT_TAGdefault:image: secopslog-py:3.15stages: [check, test, deploy]slow-lint:stage: checkscript:- sleep 15- ruff check .build:stage: checkscript:- python -m compileall -q .- echo builttest:stage: testneeds: [build] # starts as soon as build is done, without waiting for slow-lintparallel:matrix:- SUITE: [unit, smoke]script:- echo "suite $SUITE"- python -m pytest -qdocs:stage: testneeds: []rules:- changes: [docs/**/*]script:- echo "docs changed"deploy:stage: deployrules:- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCHscript:- echo "deploying $CI_COMMIT_SHORT_SHA"
Rules are checked from top to bottom, and the first one that matches decides. In workflow:rules, a match creates the pipeline, and when: never prevents it. The second rule is the one that stops duplicates: a branch push while a merge request is open for that branch creates no branch pipeline, because the merge request pipeline covers it. If no rule matches, there is no pipeline.
Job rules work the same way, per job. deploy joins only on the default branch. docs joins when files under docs/ changed. test uses needs: [build], so it starts as soon as build finishes, without waiting for the rest of its stage. And parallel:matrix turns one job definition into one job per combination of values.
A push to main
Let the runner take three jobs at once first, as in the runners lesson, so needs has room to make a difference:
Read the start times. build, slow-lint and docs started in the same second. docs has needs: [], so it didn't wait for the check stage. Both test jobs, one per matrix value, started at 03:19:32 and 03:19:33, three seconds after build finished and while slow-lint was still sleeping. deploy has no needs, so it waited for the whole test stage and also for slow-lint, and started at 03:19:45, a second after slow-lint finished. docs would have run anyway, since this push added docs/usage.md, but note the rule behind it: on the first push of a branch there is nothing to compare with, so changes is always true. To compare a new branch with main instead, write changes: { paths: [docs/**/*], compare_to: refs/heads/main }.
A feature branch
There's no deploy on the branch, as intended. Look at docs, though. The first push to feature/sub ran it although nothing under docs/ changed, and the second push skipped it. For a branch pipeline, changes compares with the previous push to that branch, and a branch's first push has nothing to compare with, so GitLab treats every file as changed. Any rule like this runs on every new branch.
A merge request
The merge request pipeline skipped docs, because in merge request pipelines changes compares the branch with the target branch, which is the comparison you actually want. And the push after the merge request opened produced exactly one pipeline, from merge_request_event. Its ref is refs/merge-requests/1/head, not feature/sub, which matters later: protected variables follow protected refs, and this ref is not one. (A project setting can let merge request pipelines receive them when both the source and target branches are protected, in the same project.)
| Variable | Branch pipeline | Merge request pipeline | Tag pipeline |
|---|---|---|---|
CI_PIPELINE_SOURCE | push | merge_request_event | push |
CI_COMMIT_BRANCH | the branch | not set | not set |
CI_COMMIT_TAG | not set | not set | the tag |
changes compares with | the previous push | the target branch | nothing: always true |
deploy to the default branch sits in a file every developer can edit on their own branch. Secrets and production access need protected branches and protected variables, from later lessons.Hands-on
Before you run it, predict which jobs a tag pipeline gets from this file. Then go back to main and push a tag: git -C app checkout main && git -C app pull && git -C app tag v0.1.0 && git -C app push origin v0.1.0, and list the jobs with QUIET=1 ./pipeline.sh app/. Were you right about docs and deploy? Finish with ./down.sh.
What you should see: the tag pipeline ran slow-lint, build, both test jobs and docs, and no deploy. deploy needs $CI_COMMIT_BRANCH, which is empty in a tag pipeline, so the rule didn't match. docs joined because a tag pipeline has no previous push to compare with, so changes is true again. If tags are your releases, give the release jobs a rule on $CI_COMMIT_TAG.
Remember
workflow:rules decides whether a pipeline exists, job rules decide who joins, and needs decides who waits for whom. Check changes rules on a branch's first push, and keep secrets out of reach of anything a rule alone protects.
workflow:rules. A developer opens a merge request and pushes again. Each push now runs the whole pipeline twice. Why?$CI_OPEN_MERGE_REQUESTS rule with when: never drops the branch pipeline while a merge request is open.needs orders jobs inside one pipeline. It never creates pipelines.rules: - changes: [docs/**/*]. A developer pushes a brand-new branch that only touches src/. Does the job run in the branch pipeline?docs ran on the first push to feature/sub and not on the second.changes rule.changes works in both. It just compares with different things.test has needs: [build], and slow-lint is in the same stage as build. With enough runner capacity, when does test start?needs. With needs, a job waits only for the jobs it names.needs turns the stage order into a graph of real dependencies.needs means "after". The job waits for build to finish.needs has nothing to do with manual jobs.