Testing, CI & supply chain: pytest, mypy, SBOM & signing

pytest/coverage, mypy, pre-commit, SBOMs and signing the tools you ship.

The tools you write have the same supply-chain obligations as any software you ship: they must be tested, type-checked, scanned, inventoried, and — for anything that runs with privilege — signed, so a consumer can verify they got the real thing. This final lesson wires the whole quality-and-provenance gate together so none of it depends on remembering to run a command.

Tests, types, and static analysis

pytest proves behaviour, coverage shows what is untested, mypy catches shape errors before runtime, and bandit flags the dangerous Python patterns (shell=True, yaml.load, pickle on untrusted data). None of these is worth much run by hand once a month; all of them are worth a lot run on every pull request as a merge gate. The point is that broken, unsafe, or untyped code cannot merge.

# run everything the gate runs, locally, before you push
pytest --cov=sec_tools --cov-fail-under=85
mypy sec_tools
bandit -r sec_tools -ll          # -ll: report medium+ severity
ruff check sec_tools             # fast lint (pyflakes + more)
 
# pre-commit ties them to the commit so nothing is forgotten
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.0
    hooks: [{ id: ruff }, { id: ruff-format }]
  - repo: https://github.com/PyCQA/bandit
    rev: 1.7.9
    hooks: [{ id: bandit, args: ["-ll"] }]

SBOM and dependency scanning

A Software Bill of Materials lists every component and version in your tool, so when the next critical CVE drops you can answer "are we affected?" in seconds instead of days. Generate an SBOM (CycloneDX or SPDX) in CI and store it with the artifact, and scan your pinned dependencies for known vulnerabilities on every build and on a schedule — a dependency that was clean at release can become vulnerable next week.

# generate a CycloneDX SBOM from the installed environment
pip install cyclonedx-bom
cyclonedx-py environment -o sbom.json
 
# scan the pinned dependencies for known CVEs (fail the build on findings)
pip-audit -r requirements.txt --strict
 
# scan the built container too — deps AND base image
trivy image --exit-code 1 --severity HIGH,CRITICAL ghcr.io/acme/sec-tools:1.2.0

Sign what you ship

For a tool that will run with access to your systems, provenance matters: consumers should be able to verify it came from your pipeline and was not swapped. Sign release artifacts and container images with cosign — keyless signing ties the signature to your CI’s OIDC identity, so there is no key to leak — and have consumers (or an admission controller) verify the signature before running. Now "is this really our tool?" has a cryptographic answer.

# sign the released image using the CI job’s OIDC identity (no private key)
cosign sign ghcr.io/acme/sec-tools:1.2.0
 
# attach the SBOM as a signed attestation
cosign attest --predicate sbom.json --type cyclonedx ghcr.io/acme/sec-tools:1.2.0
 
# consumer verifies provenance before pulling/running
cosign verify \
  --certificate-identity-regexp 'https://github.com/acme/.+' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/acme/sec-tools:1.2.0