Signing artifacts in CI

Cosign keyless from the pipeline.

Your build job pushed registry.gitlab.com/acme/api@sha256:9b2c…, the container-scanning job gave it a clean bill of health, and the deploy job shipped it. Three weeks later somebody with a stolen registry credential pushes a backdoored image into the same repository and repoints the :latest tag at it. Your scanner never sees that image. The cluster pulls :latest and runs it without a murmur. Notice what is missing from that story: any proof that the bytes running in production are the bytes your pipeline built and scanned. A courier can hand you a box with the right label on it, and only a tamper-evident seal tells you nobody opened it along the way. Signing is that seal. The pipeline cryptographically ties a signature to the exact image digest it produced (the sha256 fingerprint of those precise bytes), and a checker further down the line refuses anything that carries no signature from your pipeline's identity.

Keyless signing: the identity is the key

Classic signing hands you a private key and leaves you with a problem. The sign job has to reach that key, so the key has to live somewhere: a CI/CD variable, a key management service (KMS), a file on a runner. Wherever you park it, it becomes one more long-lived secret that can leak, and whoever steals it can sign anything they like for as long as it stays valid. Keyless signing removes the stored key entirely. It is the Sigstore model, and cosign is the tool that implements it. Rather than holding a key, the sign job proves who it is with an OpenID Connect token (OIDC, the same standard behind 'sign in with Google'), which GitLab mints fresh for that one job, and trades that proof for a certificate good for only a few minutes.

Three Sigstore services do the work. Fulcio is a certificate authority (CA, the body that vouches for who you are, like the office that issues passports). It takes the job's OIDC token, checks who issued it, and hands back a short-lived X.509 signing certificate whose subject is the CI identity written in that token. No human involved, no permanent key. Cosign then generates a throwaway key pair in memory, gets the public half certified by Fulcio, signs the digest, and discards the private half. Rekor is a public append-only transparency log, a ledger that only ever grows and never gets edited. Cosign uploads the signature and the certificate to it, so anyone can later prove the signature existed at a given moment, long after the certificate expired. The key an attacker would love to steal never outlives the job.

On the GitLab side, that token comes from an id_tokens block. You declare a token, give it the audience Fulcio expects (aud: sigstore), and GitLab drops it into the job as an environment variable. Cosign looks for a token in SIGSTORE_ID_TOKEN by itself, so naming the token that is the entire wiring job. There is no secret to configure anywhere. One caveat about older material: id_tokens replaced CI_JOB_JWT, which GitLab removed in 17.x, so any tutorial still built around CI_JOB_JWT is stale.

stages:
  - build
  - sign
 
variables:
  IMAGE: $CI_REGISTRY_IMAGE
 
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.24.0-debug
    entrypoint: [""]
  script:
    - /kaniko/executor --context "$CI_PROJECT_DIR" --dockerfile "$CI_PROJECT_DIR/Dockerfile" --destination "$IMAGE:$CI_COMMIT_SHORT_SHA" --digest-file /kaniko/digest.txt
    - echo "DIGEST=$(cat /kaniko/digest.txt)" >> build.env
  artifacts:
    reports:
      dotenv: build.env      # exposes $DIGEST to later jobs
 
sign:
  stage: sign
  image:
    name: ghcr.io/sigstore/cosign/cosign:v3.0.2
    entrypoint: [""]
  needs:
    - build
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore          # cosign reads this env var automatically
  script:
    - cosign login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
    - cosign sign --yes "${IMAGE}@${DIGEST}"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Two jobs, one dependency between them. The build job writes the digest kaniko computed into a dotenv artifact, and needs: [build] carries that DIGEST variable into the sign job, so the signature lands on the exact image this pipeline built rather than on whatever a tag happens to point at by then. The sign job needs very little else: the id_tokens block, and a registry login. Cosign stores the signature as an OCI artifact (Open Container Initiative, the standard format registries speak) sitting right next to the image in the same registry, so it needs write access, and the job token ($CI_REGISTRY_USER / $CI_REGISTRY_PASSWORD) already has that. The rules:if gate keeps signing on default-branch pipelines only. Fork pipelines and feature branches have no business minting production signatures.

$ cosign login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
Login Succeeded!
$ cosign sign --yes "${IMAGE}@${DIGEST}"
Generating ephemeral keys...
Retrieving signed certificate from the CA...
Successfully verified SCT...
tlog entry created with index: 149203847
Pushing signature to: registry.gitlab.com/acme/api
Job succeeded

Find the line tlog entry created with index: 149203847 in that log. That is your Rekor receipt. The signature and the certificate now sit in a public append-only log, and the index is a permanent pointer to the entry. Verification later leans on it to confirm the signature was made while the certificate was still valid, which is the trick that lets a certificate with a ten-minute life protect an image for months. One thing to weigh before you switch this on: Rekor is public. If your image names or digests are themselves sensitive, that metadata lands in a log anyone can read, which is why regulated organizations run their own private Sigstore infrastructure instead of the free public instances.

Pin the identity, or the check proves nothing

A signature is half a control. The other half is something that checks it before the image gets to run and fails the deploy when the check fails. Two details decide whether that check means anything at all. First, sign and verify the immutable digest (image@sha256:…), never a tag. A tag is a mutable pointer, a sticky note anyone can move, so a signature on :latest tells you nothing about the bytes :latest resolves to tomorrow. Second, pin the identity, and pin it tightly. Since cosign 2.0, leaving the identity flags off a keyless verify does not wave everything through: cosign refuses to run the check at all and stops with --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode. The risk is an identity pinned too loosely, because a pattern wide enough to match any project also matches an image an attacker signed from their own GitLab project. You have to assert two things: the OIDC issuer, and the exact certificate identity, meaning the project and ref you allow to sign.

$ cosign verify \
    --certificate-oidc-issuer "https://gitlab.com" \
    --certificate-identity "https://gitlab.com/acme/api//.gitlab-ci.yml@refs/heads/main" \
    "${IMAGE}@${DIGEST}"
 
Verification for registry.gitlab.com/acme/api@sha256:9b2c... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
[{"critical":{"identity":{"docker-reference":"registry.gitlab.com/acme/api"},"image":{"docker-manifest-digest":"sha256:9b2c..."},"type":"cosign container image signature"},"optional":{"Issuer":"https://gitlab.com","Subject":"https://gitlab.com/acme/api//.gitlab-ci.yml@refs/heads/main"}}]
$ echo $?
0

--certificate-oidc-issuer https://gitlab.com says the token behind this signature must have been minted by GitLab.com. --certificate-identity pins the full CI identity Fulcio wrote into the certificate: https://gitlab.com/acme/api//.gitlab-ci.yml@refs/heads/main, which reads as 'the .gitlab-ci.yml of the acme/api project, running on refs/heads/main'. Exit code 0 is a pass. Anything non-zero is a hard reject, no negotiation. In production you run this same command somewhere it can actually stop a rollout, and the cheapest place to start is a third job in the pipeline you already have.

stages:
  - build
  - sign
  - verify                 # the third stage
 
verify:
  stage: verify
  image:
    name: ghcr.io/sigstore/cosign/cosign:v3.0.2
    entrypoint: [""]
  needs:
    - build                # brings $DIGEST across
    - sign
  script:
    - cosign login "$CI_REGISTRY" -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
    - >-
      cosign verify
      --certificate-oidc-issuer "https://gitlab.com"
      --certificate-identity "https://gitlab.com/acme/api//.gitlab-ci.yml@refs/heads/main"
      "${IMAGE}@${DIGEST}"
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

needs: [build, sign] hands the verify job the same $DIGEST the build job wrote, so it checks the image this pipeline produced rather than whatever a tag points at, and a non-zero exit fails the stage before any deploy runs. That covers the path your own pipeline takes. It does not cover an image someone applies to the cluster by hand, so the same two flags belong in a Kubernetes admission controller as well (Sigstore Policy Controller or Kyverno), pinning the identical identity so an unsigned image, or one signed by the wrong identity, is turned away at the cluster door.

$ cosign tree "${IMAGE}@${DIGEST}"
📦 Supply Chain Security Related artifacts for an image: registry.gitlab.com/acme/api@sha256:9b2c...
└── 🔐 Signatures for an image tag: registry.gitlab.com/acme/api:sha256-9b2c....sig
   └── 🍒 sha256:3f8a1d...

cosign tree checks the plumbing without asking you to take the verify output on faith. It lists the signature artifacts the registry now holds against that exact digest, and the Rekor index from the job log can be fetched straight from the public log to cross-check. Same rule as everywhere else in this course: gate on evidence, not on 'the sign job was green'. A sign job with nothing downstream verifying its work is signing theatre.

A signature answers two questions: who built this, and has anything touched it since. It says nothing about what is inside the image, which dependencies came along for the ride, which base layers, which build produced it. Those belong to the SBOM (software bill of materials, an itemised parts list for the image) and the provenance attestation, and cosign attaches both to the very same digest and signs them the same keyless way. So the identity you just learned to pin protects those too. That is next.

Try this

Take an image your pipeline has already signed and run the verify command above on a scratch host, with --certificate-identity set to the project and branch that actually signed it. echo $? should print 0. Now run it twice more, breaking one thing each time. First change a single character in the project path inside --certificate-identity: cosign exits non-zero with none of the expected identities matched what was in the certificate and prints the subject it did find, which is the quickest way to see what your pipeline identity really looks like. Then drop --certificate-identity and --certificate-oidc-issuer altogether: cosign will not verify at all, and tells you --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode. Those three runs are your deploy gate rehearsed by hand, before you wire it into a job or an admission controller that can stop a rollout.

Takeaway

Two habits carry this whole control. Sign and verify the digest, never a tag, and pin both halves of the identity: --certificate-oidc-issuer for who minted the token, and --certificate-identity for the exact project and ref you allow to sign. Then put that verify command in front of the deploy, in the pipeline and again at the cluster door. Until something refuses to run an image that fails the check, the sign job is only writing receipts nobody reads.

Related