Container image scanning

Trivy: report all, fail on critical.

Your build job produces registry.acme.io/api@sha256:9f2a1c7e… and the next job deploys that exact digest to staging. Last Tuesday it shipped with a CRITICAL flaw in the base image's OpenSSL (the library almost every Linux program uses to encrypt network traffic), and nobody caught it, because nothing was looking between build and deploy. A port X-rays a sealed shipping container before it goes on the boat: the paperwork claims one thing, the scanner shows what is actually inside. Container image scanning is that X-ray. It reads the finished image, lists every package inside it, matches that list against the public catalogue of known CVEs (Common Vulnerabilities and Exposures, the numbered public register of disclosed security bugs), and fails the pipeline before a dangerous image gets promoted anywhere. The image is an artifact. Like every artifact in a pipeline, it gets scanned.

A container image is built in layers, the way a sandwich is: a read-only base at the bottom (a slim Debian, an Alpine, or a distroless root filesystem, meaning one stripped down to little more than your program), then everything you stack on top, an interpreter, your installed language packages, your code. That stacking leaves an image carrying two separate classes of flaw. Operating-system package CVEs live in the base: the openssl, glibc, or zlib that an apt/apk/yum layer installed. You inherited those the moment you picked a FROM line. Application-dependency CVEs live in the packages baked in above: the requests, log4j-core, or lodash your lockfile resolved and the build copied in. One scan of the finished image catches both classes, which is the whole reason you scan the image and not only the source.

Scan the digest, in a job that sits between build and deploy

stages: [build, scan, deploy]
 
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context "$CI_PROJECT_DIR"
        --destination "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
        --digest-file digest.txt
    - echo "IMAGE_DIGEST=$CI_REGISTRY_IMAGE@$(cat digest.txt)" >> build.env
  artifacts:
    reports:
      dotenv: build.env        # exports IMAGE_DIGEST to later jobs
    expire_in: 1 hour
 
container_scan:
  stage: scan
  needs: [build]
  image:
    name: aquasec/trivy:0.58.1
    entrypoint: [""]
  variables:
    TRIVY_CACHE_DIR: .trivycache
    TRIVY_NO_PROGRESS: "true"
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
  cache:
    key: trivy-db
    paths: [.trivycache]
  script:
    - trivy image --scanners vuln --severity LOW,MEDIUM,HIGH,CRITICAL --exit-code 0 "$IMAGE_DIGEST"
    - trivy image --scanners vuln --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$IMAGE_DIGEST"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

Trivy reads the image manifest, pulls each layer, and takes two inventories. First the operating-system package database (/var/lib/dpkg, /lib/apk/db, or the rpm DB), which is the receipt for everything apt or apk ever installed. Then the language manifests it finds inside the layers (Python site-packages, node_modules, jars). It matches both inventories against its own vulnerability database, a bundle of roughly 40 MB that it downloads once and, thanks to the cache: block, reuses on every later run, so the job stays under a minute. Notice that the job runs Trivy twice, on purpose. The first pass (--exit-code 0, every severity) reports everything and never fails: full visibility in the job log and in the report. The second pass is the actual gate. It runs --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed, so it fails the job only on high-impact CVEs that have a released fix somebody could apply today.

$ trivy image --severity HIGH,CRITICAL --ignore-unfixed "$IMAGE_DIGEST"
2025-01-14T09:22:31Z  INFO  Vulnerability scanning is enabled
2025-01-14T09:22:33Z  INFO  Detected OS  family="debian" version="12.8"
 
registry.acme.io/api@sha256:9f2a1c7e... (debian 12.8)
Total: 1 (HIGH: 1, CRITICAL: 0)
 
┌─────────┬───────────────┬──────────┬────────┬───────────────────┬──────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │  Fixed Version   │
├─────────┼───────────────┼──────────┼────────┼───────────────────┼──────────────────┤
│ libssl3 │ CVE-2024-6119 │ HIGH     │ fixed  │ 3.0.14-1~deb12u1  │ 3.0.14-1~deb12u2 │
└─────────┴───────────────┴──────────┴────────┴───────────────────┴──────────────────┘
 
Python (python-pkg)
Total: 1 (HIGH: 1, CRITICAL: 0)
 
┌──────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┐
│   Library    │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │
├──────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┤
│ cryptography │ CVE-2023-50782 │ HIGH     │ fixed  │ 41.0.7            │ 42.0.0        │
└──────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┘
 
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

Read the two tables and the split turns practical. The libssl3 finding sits under the OS target (debian 12.8, class os-pkgs). You fix that one by rebasing onto a patched base image or adding an apt-get upgrade layer, a platform change, usually owned by whoever maintains your base images. The cryptography finding sits under the Python target (class lang-pkgs). You fix that one by bumping a version in your lockfile, an application change, the same fix dependency scanning would have prescribed. Different owners, different remedies, one scan. And --ignore-unfixed earns its keep right here. Any sizeable base image carries CVEs with no released patch (a zlib1g marked will_not_fix, which the report-all pass still lists), and blocking merges on findings nobody can fix teaches people to switch the scanner off. Report those. Gate on the fixable HIGH/CRITICAL subset.

You might reasonably ask why you scan the image at all when dependency scanning already read the lockfile. Because the image holds what SCA (Software Composition Analysis, the scan that reads your declared dependencies) cannot see: the base image's own interpreter and system tools, a python3, a curl, a busybox, that no lockfile mentions. It also holds the exact versions that actually shipped. A stale layer cache or a floating pip install can bake in something different from what your lockfile resolved. SCA checks what you meant to ship. Image scanning checks what you really shipped. Run both. They overlap on your application's own libraries and diverge everywhere else.

The built-in template, and what actually blocks a merge

include:
  - template: Jobs/Container-Scanning.gitlab-ci.yml
 
container_scanning:
  stage: scan
  needs:
    - job: build
      artifacts: true          # inherits IMAGE_DIGEST from build's dotenv
  variables:
    CS_IMAGE: "$IMAGE_DIGEST"   # scan the exact digest, not a floating tag
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

GitLab ships a maintained template for this, Jobs/Container-Scanning.gitlab-ci.yml. It wraps the same Trivy engine, and the part that matters is what comes out the far end: a structured container_scanning report that GitLab renders in the merge-request security widget and in the Vulnerability Report. Point CS_IMAGE at your digest and it scans for you. One catch trips nearly everyone. The template does not fail the pipeline on findings by default. It reports them. To block a merge you attach a merge-request approval policy (formerly called a scan result policy) that demands an approval whenever a container_scanning finding is HIGH or above. So you have two workable gates. Run trivy image --exit-code 1 yourself and fail the job: available on any tier, blunt, decided inside the pipeline. Or run the template plus a policy and gate on the evidence the platform already collected: Ultimate only, and auditable. Pick one. A template left running with no policy attached is how two HIGH findings sail through a pipeline that looks green.

# Verify the finding split programmatically (os-pkgs vs lang-pkgs):
$ trivy image --format json --severity HIGH,CRITICAL --ignore-unfixed "$IMAGE_DIGEST" \
    | jq -r '.Results[] | select(.Vulnerabilities) | "\(.Target) [\(.Class)]: \(.Vulnerabilities|length)"'
registry.acme.io/api@sha256:9f2a1c7e... (debian 12.8) [os-pkgs]: 1
Python [lang-pkgs]: 1
 
# Verify the gate actually failed the job, via the GitLab Jobs API:
$ glab api "projects/$CI_PROJECT_ID/jobs/$CI_JOB_ID" | jq '{name, status}'
{
  "name": "container_scan",
  "status": "failed"
}

Every check in this lesson reads the image while it sits still, its packages and layers matched against bugs somebody already found and numbered. The next lesson flips that around: DAST (Dynamic Application Security Testing) and the scan pipeline, where the application gets deployed and then poked at while it runs, catching the flaws that only exist once the code is executing and that no reading of the layers could ever reveal.

Try this

Run trivy image --severity HIGH,CRITICAL --ignore-unfixed "$IMAGE_DIGEST" on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: scan the digest, never the tag. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related