CI/CD variables, masking & secret managers

Secrets that do not leak into logs or forks.

Your deploy job has to run docker login and push the image it just built, so it needs a registry password. Where does that password live? Not in the repo. Committing a secret to Git is like taping the house key to the front door: every clone, every fork, and every commit in the history carries it from that moment on, and deleting it later does not take any of that back. GitLab's answer is the CI/CD variable (CI/CD is continuous integration and continuous delivery, the automated build-test-ship pipeline). You set the value once in Settings → CI/CD → Variables, or through the API, and GitLab hands it to your jobs as an ordinary environment variable. That gets the secret out of the repo, which is one problem out of three. The value still lands in the job log the second a script echoes it. And by default GitLab will hand it to pipelines built from a feature branch or a fork, meaning code nobody has reviewed. Two per-variable flags close those two holes. A third pattern removes the stored password altogether. All three are this lesson.

Two checkboxes, two different jobs

Masked and protected sit next to each other in the same settings form, they sound like synonyms, and people mix them up constantly. They do completely different work. Masked is the bleep on live television: GitLab watches the job output and swaps the value for [MASKED] anywhere it appears, so an accidental echo, or a chatty tool that prints its own arguments, does not put the secret on screen. Protected is the guest list at the door: the variable only goes to jobs running on a protected branch or a protected tag, so a pipeline from a fork or a random feature branch never receives it at all. Masking is about the log. Protection is about which pipelines get the value in the first place. A production secret wants both, every single time. Masking also comes with format rules the value has to satisfy: one line, no spaces, at least eight characters, and built only from characters GitLab can match reliably (letters and digits plus a short list of symbols such as -, _, @, ., and ~). The reason is unglamorous. GitLab can only redact a literal string it is able to find in the output stream, so a value it cannot pattern-match is a value it cannot hide.

$ curl --request POST \
    --header "PRIVATE-TOKEN: $ADMIN_PAT" \
    "https://gitlab.acme.internal/api/v4/projects/42/variables" \
    --form "key=REGISTRY_TOKEN" \
    --form "value=glpat-3xAmpLe7k9Qv2WdN0pLz" \
    --form "masked=true" \
    --form "protected=true"

{
  "variable_type": "env_var",
  "key": "REGISTRY_TOKEN",
  "value": "glpat-3xAmpLe7k9Qv2WdN0pLz",
  "protected": true,
  "masked": true,
  "hidden": false,
  "raw": false,
  "environment_scope": "*",
  "description": null
}

Creating the variable through the API pays off immediately: the response hands the flags straight back, protected: true and masked: true, so a setup script or a reviewer can assert them in code instead of trusting that somebody ticked the right box in a web form. That matters once a project has forty variables and one of them is a box nobody ticked. You cannot grep a checkbox. Now watch what masking actually does at runtime. Here is a throwaway job that references the variable, and the log where the redaction lands.

show-registry:
  stage: build
  script:
    - echo "authenticating to $CI_REGISTRY as ci with token $REGISTRY_TOKEN"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

$ echo "authenticating to $CI_REGISTRY as ci with token $REGISTRY_TOKEN"
authenticating to registry.acme.internal as ci with token [MASKED]
Cleaning up project directory and file based variables
Job succeeded

That [MASKED] is a relief to see, and worth having, but masking is best effort and it only ever matches the exact literal value. Change the bytes before they print (base64-encode the secret, chop it in half, uppercase it, wrap it in JSON) and the transformed version sails into the log untouched, because it no longer equals the string GitLab is watching for. Masking also stops at the edge of the log. A token written into an artifact, dropped into a cache entry, or caught by an env dump is plain text to anyone who can read the pipeline. So treat masking as a seatbelt, not a plan. Never print a secret on purpose, never write one to a file that gets uploaded, and assume a careless script (or a hostile one) can beat the redaction whenever it feels like it.

The better pattern: nothing left to steal

A masked, protected variable is still a standing credential parked in GitLab. Miles better than a committed one, but it is a thing you have to store, rotate, and worry about. The newer pattern gets rid of it. A spare office key you hand out once and hope comes back is one model. A visitor badge printed at reception after the guard checks your ID, dead by five o'clock, is the other. OpenID Connect (OIDC, a standard way for one system to prove who it is to another using a signed token) is the visitor badge. The job proves its identity to your secret manager and gets back a short-lived token, so nothing durable sits in GitLab. GitLab issues the proof through the id_tokens keyword. For each token you name, you declare an aud (audience, meaning who the token is intended for), and GitLab mints a JWT (JSON Web Token, a small signed blob of facts) for that one job, carrying claims a verifier can check: project_id, ref, ref_protected, user_login. HashiCorp Vault's JWT auth method checks that signature against GitLab's published signing keys (JWKS, the JSON Web Key Set, the public keys GitLab publishes so anyone can verify what it signed), matches the claims against a role you configured, and returns a Vault token carrying exactly the policies that role allows and nothing beyond them.

Trust gets set up once, on the Vault side, and this is the load-bearing part of the whole design: the role decides which pipelines Vault is willing to believe. You point a JWT auth role at GitLab as the token issuer, then pin bound_claims so that only the project you meant, and only its protected-branch jobs, can authenticate as that role. Bind the audience as well, so a token minted for some other service cannot be replayed here.

$ vault write auth/jwt/config \
    oidc_discovery_url="https://gitlab.acme.internal"
 
$ vault write auth/jwt/role/gitlab-ci \
    role_type="jwt" \
    user_claim="user_login" \
    bound_audiences="https://vault.acme.internal" \
    bound_claims_type="string" \
    bound_claims='{"project_id":"42","ref":"main","ref_protected":"true"}' \
    token_policies="registry-read" \
    token_ttl="5m"

Success! Data written to: auth/jwt/config
Success! Data written to: auth/jwt/role/gitlab-ci

Now the pipeline side. The id_tokens block declares VAULT_ID_TOKEN with the audience Vault expects. The script trades that JWT for a short-lived Vault token, reads the registry password out of Vault, and pipes it straight into docker login. The password lives for the length of one job and is never stored in GitLab at all. (GitLab Premium and Ultimate can express the same fetch declaratively with the secrets: keyword, but the id_tokens plus jwt-login form below runs on every tier and shows you exactly what is happening underneath.)

deploy:
  stage: deploy
  image: registry.acme.internal/tools/deployer:1.4
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://vault.acme.internal
  variables:
    VAULT_ADDR: https://vault.acme.internal
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=gitlab-ci jwt=$VAULT_ID_TOKEN)"
    - vault kv get -field=token secret/ci/registry | docker login -u ci --password-stdin "$CI_REGISTRY"
    - docker push "$CI_REGISTRY/acme/app:$CI_COMMIT_SHA"

$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=gitlab-ci jwt=$VAULT_ID_TOKEN)"
$ vault kv get -field=token secret/ci/registry | docker login -u ci --password-stdin "$CI_REGISTRY"
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Login Succeeded
$ docker push "$CI_REGISTRY/acme/app:$CI_COMMIT_SHA"
5f2e3a1c9b0d: Pushed
2c1a0f8e7d6b: Pushed
Job succeeded

So which mechanism for which secret? A value committed to the repo is never acceptable, at any team size. A masked, protected CI/CD variable is a fair baseline for a low-value credential, or one that is genuinely painful to rotate, on a single project. OIDC into Vault is the target for anything production-grade, and a cloud secrets manager counts too, because AWS, Google Cloud, and Azure all accept a GitLab id_token the same way. The credential is short-lived, minted for one job, rotated in one central place, and every fetch lands in Vault's audit log, which happens to be exactly the evidence a deploy gate or an incident review asks for. It scales, too. One Vault role serves dozens of pipelines, and cutting off access is a single policy change instead of a hunt through every project that ever pasted the token somewhere. The cost is real: you need a Vault or a cloud IAM (identity and access management, the service that decides who may do what) to run, and GitLab's JWKS endpoint has to be reachable from whatever is doing the verifying. For a two-person side project, a protected variable may well be enough. The moment a secret is shared across teams, worth stealing, or subject to audit, federate it.

Forks are where this gets sharp in practice. GitLab does not pass protected variables to a pipeline running in a forked project, which is the behavior you want. But a merge request from a branch inside your own project runs on your pipeline, with whatever that ref is allowed to see, so protected only saves you if the branch really is protected. That is the same trust boundary as the runner split from the previous lesson, seen from a different angle. OIDC shrinks the exposure further. Even if an unreviewed job talks GitLab into minting an id_token, Vault's bound_claims throw it straight out unless the claims name your project and a protected ref, so the token is dead weight anywhere off the trusted path. That is the appeal in one line: the defense stops depending on anyone remembering to tick a box, or on a script behaving itself.

Try this

Work through “The better pattern: nothing left to steal” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a Vault role with no bound_claims trusts your whole GitLab. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related