>_SecOpsLog_
CoursesSoftware supply chain securityContinuous rescanning & VEX

Continuous rescanning & VEX

Yesterday’s clean image is today’s CVE.

Advanced12 min · lesson 16 of 18

A recall notice is a strange kind of danger. The car in your driveway is the same machine it was yesterday. Nothing about it changed. But this morning the maker announced that one of its airbags can misfire, and now that unchanged car is a hazard. The car didn't get worse. The world's knowledge got better, and the car crossed from 'fine' to 'known problem' without moving an inch.

A container image you shipped works the same way. You built it, scanned it, got a clean bill of health, signed it, and pushed it to production. Three weeks later a researcher publishes a serious flaw in a library baked inside that image. Not one byte of your artifact moved. Its security posture still fell off a cliff, because 'clean' only ever meant 'no publicly known vulnerabilities on the day I looked.' New ones are disclosed every day, against code you already shipped. So the honest question is never 'is this image secure?' It is 'is this image secure as of right now?', and the only way to answer it is to keep asking.

You already have the parts list

When you built that image you generated an SBOM (Software Bill of Materials, an itemized inventory of every library, binary, and version inside the artifact, the way a food label lists every ingredient). Keep those SBOMs somewhere you can query them. That saved list is what makes rescanning nearly free: you are not rebuilding the image or even pulling it from the registry, you are re-checking a stored ingredient list against a fresh list of recalls. The tool that does the matching is grype (an open-source scanner from Anchore that reads an SBOM and compares each component against a database of known vulnerabilities). A CVE (Common Vulnerabilities and Exposures identifier, a unique public serial number for one specific security flaw so everyone refers to it the same way) is one line item in that recall list.

Here is the part that surprises people. The same SBOM, unchanged, returns more findings as time passes, because grype checks it against today's data, not the day-you-built data. Run it by hand once so you can read a result before you automate it.

terminal
# Re-check a stored SBOM against today's vulnerability data (no rebuild, no image pull)
grype sbom:./inventory/payments-api.spdx.json --only-fixed
output
 ✔ Vulnerability DB                [no update available]
 ✔ Scanned for vulnerabilities     [3 vulnerability matches]
   ├── by severity: 1 critical, 1 high, 1 medium, 0 low, 0 negligible
   └── by status:   3 fixed, 0 not-fixed
 
NAME       INSTALLED                 FIXED-IN                  TYPE  VULNERABILITY    SEVERITY
libcurl4   7.81.0-1ubuntu1.6         7.81.0-1ubuntu1.15        deb   CVE-2023-38545   Critical
openssl    3.0.2-0ubuntu1.6          3.0.2-0ubuntu1.7          deb   CVE-2022-3602    High
libxml2    2.9.13+dfsg-1ubuntu0.1    2.9.13+dfsg-1ubuntu0.4    deb   CVE-2024-25062   Medium

The day you built this, that same file returned zero matches. Today it returns three, and the image never changed. Each row names the component, the version you are running, the version that fixes it, the package type, and the CVE. The --only-fixed flag trims the list to vulnerabilities that already have a fix upstream, which is the set you can act on tonight. That single scan is a snapshot. To operate, you need it to happen on its own.

Put it on a timer

Running a scan by hand once proves nothing about tomorrow. You want the rescan to walk the same rounds every night whether or not anyone tells it to, like a night-shift guard. On a current Linux box that guard is a systemd timer (systemd is the init system that starts and supervises background services on most modern distributions; a timer is its built-in scheduler, a cleaner replacement for cron, the classic Unix job scheduler). Three small files set it up: a script that does the work, a service that runs the script, and a timer that decides when.

/usr/local/bin/rescan.sh
#!/usr/bin/env bash
set -euo pipefail
 
INV=/var/lib/secops/inventory     # stored SBOMs, one per running image
VEX=/var/lib/secops/vex           # OpenVEX docs holding our triage decisions
rc=0
 
grype db update                   # pull the newest vulnerability data first
 
for sbom in "$INV"/*.spdx.json; do
  name=$(basename "$sbom" .spdx.json)
  echo "== $name =="
  vexdoc="$VEX/$name.vex.json"
  args=(--fail-on critical --only-fixed --show-suppressed)
  [[ -f "$vexdoc" ]] && args+=(--vex "$vexdoc")
  grype "sbom:$sbom" "${args[@]}" || rc=1   # keep scanning the rest, remember the failure
done
 
exit "$rc"        # non-zero -> systemd marks the unit failed -> OnFailure fires
/etc/systemd/system/rescan.service
[Unit]
Description=Rescan running images against fresh CVE data
Wants=network-online.target
After=network-online.target
OnFailure=rescan-alert.service     # a fixable critical pages on-call
 
[Service]
Type=oneshot
ExecStart=/usr/local/bin/rescan.sh
Nice=10
IOSchedulingClass=idle             # stay out of the way of production load
/etc/systemd/system/rescan.timer
[Unit]
Description=Nightly rescan of running images
 
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=20min           # don't hit the DB mirror at the same second fleet-wide
Persistent=true                    # if the box was off at 02:30, run at next boot
 
[Install]
WantedBy=timers.target

A few lines are doing real work here. grype db update at the top pulls the newest vulnerability data before scanning, so 'fresh' actually means fresh (a rescan against a stale database is theater). --fail-on critical makes grype exit non-zero when it finds a fixable critical, which fails the systemd service, which fires OnFailure=rescan-alert.service (a companion unit that pages whoever is on call). Persistent=true catches up a missed run if the machine was down at 02:30. RandomizedDelaySec smears the load so a fleet of hosts does not stampede the same database mirror at once. Enable it and confirm the schedule.

terminal
sudo systemctl daemon-reload
sudo systemctl enable --now rescan.timer
systemctl list-timers rescan.timer
output
Created symlink /etc/systemd/system/timers.target.wants/rescan.timer → /etc/systemd/system/rescan.timer.
 
NEXT                        LEFT      LAST  PASSED  UNIT          ACTIVATES
Sat 2026-07-18 02:38:11 UTC 15h left  n/a   n/a     rescan.timer  rescan.service
 
1 timers listed.

A freshly enabled timer has never fired, so LAST and PASSED read n/a until 02:30 rolls around. That covers the routine. Some days are not routine. When a headline vulnerability drops at noon, you do not wait for 02:30, you run the same job now and watch the log with journalctl (the command that reads the systemd journal, the system's central log).

terminal
# A big CVE just dropped; scan right now instead of waiting for tonight
sudo systemctl start rescan.service
journalctl -u rescan.service -u rescan-alert.service -n 12 --no-pager
output
Jul 17 12:04:48 ops-01 systemd[1]: Starting rescan.service - Rescan running images against fresh CVE data...
Jul 17 12:04:49 ops-01 rescan.sh[24187]: == billing-worker ==
Jul 17 12:04:51 ops-01 rescan.sh[24187]: [0000]  WARN discovered vulnerabilities at or above the severity threshold
Jul 17 12:04:51 ops-01 rescan.sh[24187]: == payments-api ==
Jul 17 12:04:53 ops-01 systemd[1]: rescan.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 12:04:53 ops-01 systemd[1]: rescan.service: Failed with result 'exit-code'.
Jul 17 12:04:53 ops-01 systemd[1]: rescan.service: Triggering OnFailure= dependencies.
Jul 17 12:04:53 ops-01 systemd[1]: Starting rescan-alert.service - Page on-call about a critical finding...
Jul 17 12:04:53 ops-01 systemd[1]: Finished rescan-alert.service - Page on-call about a critical finding.

That is the whole detection story on one screen. An image you have not touched in weeks trips a fixable critical, the service exits non-zero, and systemd hands off to your alerting unit. No human had to remember to check. The catch is what comes next, because if you leave it here, you will drown.

Cut the noise, or the noise cuts you

Rescan nightly and you hit one guaranteed side effect: noise. A scanner reports what could be wrong, not what is wrong for you. It sees that your image ships libcurl 7.81 and that this version has a critical heap overflow (CVE-2023-38545), and it fires. What it cannot see is that the overflow only triggers when libcurl connects through a SOCKS5 proxy (an older kind of network middleman that forwards your connections on your behalf) using a very long hostname, and your service never uses a SOCKS5 proxy at all. The alarm is technically correct and operationally meaningless. A smoke detector that shrieks every time you make toast gets taped over, and then it will not warn you about a real fire.

VEX (Vulnerability Exploitability eXchange, a standard, machine-readable format for recording a human's judgment about whether a given vulnerability actually affects a given product) is how you write 'this alarm is expected, here is exactly why, do not page me for it' in a form a scanner can honor and an auditor can check months later. A VEX statement carries a status: one of not_affected, affected, fixed, or under_investigation. When you claim not_affected, the format makes you attach a justification from a fixed list, so you cannot wave the finding away with a shrug. vulnerable_code_not_in_execute_path means the flawed code is present but never runs. component_not_present means it is not even there. The point is to force a true reason and record it where the next rescan can read it.

That libcurl critical is real code but unreachable for you: the service makes plain HTTPS calls (ordinary encrypted web traffic) and never enables SOCKS5 proxying, so the vulnerable handshake can never be hit. Textbook not_affected. You can hand-write the statement or generate it with vexctl create --product 'pkg:oci/payments-api@sha256:9f2a1c8b...' --vuln CVE-2023-38545 --status not_affected --justification vulnerable_code_not_in_execute_path (vexctl is the OpenVEX command-line tool). Either way you commit a document that looks like this, then enrich it with a plain-English impact statement:

vex/payments-api.vex.json
{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://secopslog.com/vex/payments-api/2026-07-17",
  "author": "SecOps <[email protected]>",
  "timestamp": "2026-07-17T12:05:11Z",
  "version": 1,
  "statements": [
    {
      "vulnerability": { "name": "CVE-2023-38545" },
      "products": [
        { "@id": "pkg:oci/payments-api@sha256:9f2a1c8b..." }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path",
      "impact_statement": "libcurl is present but the service never uses a SOCKS5 proxy; the overflow in the SOCKS5 handshake is unreachable."
    }
  ]
}

The product is named by a purl (package URL, a standard string that points at one exact package; here pkg:oci/... names an OCI image, the Open Container Initiative's image format) locked to a sha256 digest. That pin matters: the statement is a claim about this exact build, not about 'payments-api' in general. Now feed the VEX doc back into the scan and watch what happens to the gate.

terminal
grype sbom:./inventory/payments-api.spdx.json \
      --vex vex/payments-api.vex.json \
      --fail-on critical --only-fixed --show-suppressed
echo "exit: $?"
output
 ✔ Vulnerability DB                [no update available]
 ✔ Scanned for vulnerabilities     [2 vulnerability matches]
   ├── by severity: 0 critical, 1 high, 1 medium, 0 low, 0 negligible
   └── by status:   2 fixed, 0 not-fixed
 
NAME       INSTALLED                 FIXED-IN                  TYPE  VULNERABILITY    SEVERITY
libcurl4   7.81.0-1ubuntu1.6         7.81.0-1ubuntu1.15        deb   CVE-2023-38545   Critical   (suppressed by VEX)
openssl    3.0.2-0ubuntu1.6          3.0.2-0ubuntu1.7          deb   CVE-2022-3602    High
libxml2    2.9.13+dfsg-1ubuntu0.1    2.9.13+dfsg-1ubuntu0.4    deb   CVE-2024-25062   Medium
exit: 0

Same three findings, but the critical is now tagged (suppressed by VEX) and pulled out of the counts that --fail-on gates on. The match count drops to two, the critical count reads zero, and the job exits 0. Nobody gets paged at 2am for a vulnerability you already reasoned about in daylight. The finding did not vanish (with --show-suppressed it is right there, your justification attached); it stopped being an alarm. That is signal preserved, noise removed, with a paper trail.

From alarm to fix

Suppression is one of two exits from triage. The other is that the finding is real and reachable, and then it must drive a change. The OpenSSL High in that same scan (CVE-2022-3602, a 4-byte overflow in the code that decodes names inside X.509 certificates, the digital ID cards that prove a server is who it claims to be) is the reachable kind. Your service checks a certificate on every outbound HTTPS call it makes, using that same library to negotiate TLS (Transport Layer Security, the encryption layer under HTTPS), so the certificate-parsing code runs constantly. That one earns the update path you built earlier: bump the dependency or rebuild against a patched base image, regenerate the SBOM, rescan, re-sign, redeploy, on a clock set by severity (a reachable critical measured in hours, a high in days, whatever your policy states). Deciding reachability is the hard, human part. Call-graph and static-analysis tools can help, but someone has to make the call, and when they say 'not affected,' they write it down as VEX so tomorrow's rescan does not relitigate it.

A VEX statement can quietly start lying for you
not_affected is a claim about one image digest at one moment. The day a refactor starts routing attacker-controlled input into that libcurl SOCKS5 path, the vulnerability becomes reachable and your old statement is a false 'all clear' that your scanner now trusts. Pin every VEX statement to a digest, re-verify the justification on each release, and never copy a statement forward to a new build without checking that it still holds.

One caution about the --only-fixed flag you have been leaning on. It drops every vulnerability that has no upstream patch yet, which is what keeps the nightly page short and actionable. It is not a license to forget those. A critical with no fix can still demand a config change, a network control in front of the service, or pulling the component out entirely. Run the scan without --only-fixed on its own slower cadence, weekly say, so the no-fix-yet pile stays visible and owned by a name.

The rescan-to-action loop that never stops
1Stored SBOM
per running image, saved at build
2Scheduled rescan
nightly timer + on big-CVE days
3Fresh CVE match
image unchanged, database changed
4Triage: reachable?
can the vulnerable code actually run
5VEX or patch
not_affected suppresses; affected drives a fix
6Re-sign & redeploy
only for real fixes, then loop repeats
Every night the same image is re-judged against a world that learned something new.
Quick check
01The nightly rescan flags CVE-2023-38545 against payments-api, and the stored SBOM confirms libcurl 7.81.0-1ubuntu1.6 is installed. Which of these, if true, would hold up as the justification behind a not_affected statement?
Incorrect — A published fix tells you what to install and nothing more. Until that build is signed and deployed you are still running the flawed version, so the honest status is affected with a clock on it.
Correct — This is what the vulnerable_code_not_in_execute_path justification claims. The flawed code ships inside the image, but no input the service handles can arrive at it, so the statement survives an auditor reading it months later.
Incorrect — A clean scan only covered the flaws known that morning. Later matches against an unchanged image are the entire reason you rescan, not evidence that the tool has gone wrong.
Incorrect — VEX makes you pick a justification from a fixed list precisely so convenience cannot be the reason. Writing one to keep a job green is taping over the smoke detector, and the paper trail records that you did.
02rescan.sh calls grype db update once before it loops over $INV/*.spdx.json. A teammate wants to delete that line to shave a minute off the nightly run. What would you lose?
Incorrect — A rescan never touches the registry or the image. It re-reads a parts list you saved at build time, and that update line has nothing to do with regenerating it.
Incorrect — grype db update refreshes the vulnerability data the scanner matches against. Upgrading the program itself is a package manager job on your own schedule, not something the timer does.
Correct — The SBOM is byte-identical between runs, so every new finding has to come from the database learning something overnight. Skip the refresh and the job still scans, still reports, and still tells you exactly what it told you yesterday.
Incorrect — Severity comes from the same data the scan already reads, stale or fresh. An out-of-date database still rates what it recognises, which is why a stale run can look reassuring while missing this week's disclosures.
03The nightly job has exited 0 every night this week, yet one shipped image carries a critical with no upstream patch that never reached the page. What happened, and what do you change?
Correct — The flag keeps the 2am page down to work someone can actually do tonight. A finding you cannot patch can still need a config change, a network control in front of the service, or the component pulled out, so it needs its own slower cadence.
Incorrect — Suppression only happens where someone wrote a statement pinned to a digest with a justification attached. --show-suppressed is already in the args too, so a VEX-silenced row would still print, tagged.
Incorrect — The scanner knows about these; its summary line counts them under not-fixed. Your flag is what removed them from the table, and dropping the flag brings them straight back.
Incorrect — The script already refreshes before it scans anything, and one refresh covers every SBOM in the loop. Repeating it per image costs time and changes nothing about what the filter is hiding.

Set the timer tonight. The first time a false alarm wakes you, write the VEX statement in daylight, pin it to the digest, and drop a reminder on your calendar to re-check that justification at the next release. Everything you shipped is quietly aging toward its own recall notice, and the only way to hear about it is to keep asking. The night the rescan fails is the night it earned its keep.

Try this

Run grype sbom:./inventory/payments-api.spdx.json --only-fixed on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a VEX statement can quietly start lying for you. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Registry trust & allowed sourcesSupply-chain incident response

Related

What CI is, and where Jenkins fitsJenkins foundations, done rightThe .gitlab-ci.yml modelSecure CI/CD with GitLabSupply-chain threat modelSoftware supply chain in depthCI/CD interview questionsInterview prep