Scan images with Trivy

Report everything, fail only on criticals.

A container image is a stack of tarballs, and old software hides in the lower layers. You inherit those layers. The base you picked six months ago shipped with a version of OpenSSL that has grown new vulnerabilities since, and every image built on top of it carries the same rot forward. Nobody touched the file. Nobody rebuilt. It aged in place while the CVE feeds kept moving. Scanning is how you find that rot before it reaches a cluster, instead of after someone else does.

Trivy is a customs X-ray for images. One static binary, no server to run, and a single pass that reads the operating-system packages, your language lockfiles, and even secrets someone baked into a layer by accident. It checks misconfigurations too, but not on an image unless you ask: add --scanners misconfig, and even then it only reads infrastructure-as-code files that happen to be sitting inside the image. Each finding maps to a CVE (Common Vulnerabilities and Exposures) identifier with a severity, and when a patch exists, the version that fixes it. Run it in CI (continuous integration), on every push, not only in the registry. The reason is the feedback loop. A HIGH caught in a merge request is a ten-minute bump of a base image. The same HIGH found in production is a change ticket, a rollout, and a line in a postmortem.

$ trivy image registry.internal/payments-api:1.4.2
payments-api:1.4.2 (debian 12.5)
================================
Total: 23 (UNKNOWN: 0, LOW: 14, MEDIUM: 7, HIGH: 2, CRITICAL: 0)
 
┌───────────────┬────────────────┬──────────┬───────────────────┐
│    Library    │ Vulnerability  │ Severity │  Fixed Version    │
├───────────────┼────────────────┼──────────┼───────────────────┤
│ libssl3       │ CVE-2024-6119  │ HIGH     │ 3.0.14-1~deb12u2  │

Read that table from right to left. The Fixed Version column is the one that decides your day: if it's filled in, someone upstream already did the work and you just rebuild on the newer base. Severity tells you how loud to be about it. Library and Vulnerability tell you exactly what to blame.

Trivy is really reading two different worlds in that single pass, and they fail in different ways. Operating-system packages (libssl3, glibc, the apt-installed plumbing) get fixed by rebuilding on a patched base image or bumping the package. Your application dependencies, the code inside a package-lock.json or a Gemfile.lock or a fat jar, get fixed by editing a line in your own manifest and rebuilding. Same table, two fix paths. When the Fixed Version names an OS package, the move is almost always to pull a newer base tag and rebuild. When it names a library you pulled in yourself, the fix is yours to own.

Fail on what matters, not everything

A scanner that blocks every merge gets switched off inside a month. Someone comments it out at 2am to ship a hotfix, and it never comes back. So split the job in two. The first pass is a bouncer writing down every name at the door and waving everyone through: exit code 0, nothing fails. The second pass turns away only the people on the real ban list: exit code 1, and only on the severities you'd genuinely wake up for. Add --ignore-unfixed so you stop blocking on CVEs that have no patch yet. A developer can't fix what the upstream maintainer hasn't fixed, and a gate nobody can satisfy is a gate people learn to route around.

container_scan:
  image: { name: aquasec/trivy:0.53.0, entrypoint: [""] }   # pin the scanner
  script:
    - trivy image --exit-code 0 --severity LOW,MEDIUM,HIGH,CRITICAL "$IMAGE"
    - trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed "$IMAGE"

Prove the gate actually bites. Run the blocking pass by hand against an image you know is dirty, and check the exit code, because a job that prints findings but always exits 0 is decoration, not a control.

$ trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed \
    registry.internal/legacy-auth:0.9
legacy-auth:0.9 (jar)
Total: 1 (CRITICAL: 1)
 
│ log4j-core │ CVE-2021-44228 │ CRITICAL │ 2.15.0 │
$ echo $?
1                     # non-zero exit fails the GitLab job, the merge is blocked

The flag earns its keep. Watch what --ignore-unfixed does to the same image: the noise you can't act on drops out, and what's left is a shortlist a developer can clear today.

$ trivy image -q --severity HIGH,CRITICAL \
    registry.internal/payments-api:1.4.2 | grep Total
Total: 2 (HIGH: 2, CRITICAL: 0)
$ trivy image -q --severity HIGH,CRITICAL --ignore-unfixed \
    registry.internal/payments-api:1.4.2 | grep Total
Total: 1 (HIGH: 1, CRITICAL: 0)     # one HIGH has no upstream fix yet, dropped

Accept a risk on purpose, not by accident

--ignore-unfixed handles the CVEs with no patch. But every so often you hit one that does have a fix and you still don't want it failing the build tomorrow. You've read it, the exploit needs a code path you never call, and you've decided to carry the risk until the next base-image bump. Suppressing it is a real decision, so make it a visible one. A .trivyignore file is a plain list of CVE IDs you've triaged, one per line; Trivy reads it and skips exactly those. Commit it to the repo, so the suppression is reviewable and blames back to a person like any other change.

# .trivyignore: reviewed exceptions, each one signed off by a person
 
# libssl3: the vulnerable TLS-client path is never reached, tracked in SEC-412
CVE-2024-6119
 
# zlib/minizip: not linked in our build, upstream marked it wontfix
CVE-2023-45853

Two habits keep that file honest. Give every entry a comment saying why, so the next reviewer isn't reverse-engineering your reasoning six months from now. And treat every exception as temporary by default. Trivy's YAML ignore format lets you set an expired_at date, so the suppression re-arms itself the moment the deadline passes and the CVE comes back to bite if nobody dealt with it. An ignore file with no reasons and no dates isn't risk management. It's a quiet place where CRITICALs go to be forgotten.

Scan what is already running

CI covers what you're about to ship. It says nothing about what shipped last quarter and has been running ever since. So point Trivy at the images a cluster is running right now. Pull the image list straight from the pods, scan each one, and read the totals to triage. This is the fast answer to the question a fresh zero-day forces on you at 9am: what are we actually running that's vulnerable? Cache the vulnerability database, roughly 40 MB, so repeated scans stay quick.

# custom-columns joins a multi-container pod's images with commas, so split on both
$ kubectl get pods -A \
    -o custom-columns="IMAGE:.spec.containers[*].image" --no-headers \
    | tr -s ' ,' '\n' | sort -u \
    | while read img; do echo "== $img"; trivy image -q -s HIGH,CRITICAL "$img"; done
== registry.internal/envoy-sidecar:1.29
Total: 3 (HIGH: 3, CRITICAL: 0)     # a sidecar, only listed because the split handles commas
== registry.internal/legacy-auth:0.9
Total: 6 (HIGH: 5, CRITICAL: 1)     # the log4j box; page on this one first
== registry.internal/payments-api:1.4.2
Total: 2 (HIGH: 2, CRITICAL: 0)

That one-liner reads .spec.containers, which means it quietly skips init and ephemeral containers. On a real audit, widen the jsonpath to catch those too, because an init container that runs once at startup still pulled a full image onto your node. Trivy also ships a native trivy k8s mode that talks to the API server directly and scans what it finds, which is the cleaner option for a scheduled cluster-wide sweep. The kubectl loop is the one you can read top to bottom and trust, so it's the one to reach for when you're triaging at 9am and need an answer you can defend.

Tag mutation means yesterday's clean :latest is not today's. Prefer digest pins in Deployments after the scan passes.

Runtime scans catch drift and sidecars you forgot to rebuild. Schedule them; do not wait for an audit.

Noise kills programs. Tune severities and paths so on-call sees actionable CVEs, not a thousand info findings.

Separate build-time and deploy-time responsibilities. Build CI fails on new CRITICAL in the app image. A cluster scanner fails on CRITICAL that appears in running pods after a base rebuild. Both tickets should link to the digest, not the tag. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Scan an image with Trivy, fail on CRITICAL, and record one accepted CVE with an owner and expiry.

$ trivy image --severity CRITICAL --exit-code 1 registry.internal/legacy-auth:0.9
...
Total: 1 (CRITICAL: 1)
$ echo exit:$?
exit:1
 
$ cat .trivyignore.yaml
vulnerabilities:
  - id: CVE-2024-6119
    statement: "owner=payments-team, TLS-client path unreachable, tracked in SEC-412"
    expired_at: 2026-08-01
 
$ trivy image -q --severity HIGH,CRITICAL --ignorefile .trivyignore.yaml \
    registry.internal/payments-api:1.4.2 | grep Total
Total: 1 (HIGH: 1, CRITICAL: 0)     # the dated exception drops one; it re-arms on 2026-08-01
 
# pinned scanner is 0.53; the trivy k8s targets and flags shift between releases, check --help
$ trivy k8s -n payments --report summary all
Summary Report for prod-eu
 
Workload Assessment
┌───────────┬─────────────────────────┬───────────────────┐
│ Namespace │        Resource         │  Vulnerabilities  │
│           │                         ├───┬───┬───┬───┬───┤
│           │                         │ C │ H │ M │ L │ U │
├───────────┼─────────────────────────┼───┼───┼───┼───┼───┤
│ payments  │ Deployment/payments-api │ 0 │ 2 │ 7 │14 │ 0 │
│ payments  │ Deployment/legacy-auth  │ 1 │ 5 │ 3 │ 9 │ 0 │
└───────────┴─────────────────────────┴───┴───┴───┴───┴───┘
# abridged: the real report also carries Misconfigurations and Secrets column groups

Takeaway

Scan at build and again in the cluster. Fail on severities you actually triage, and make every ignore a dated exception.

Related