Packaging, pinning and auditing your tool

pyproject.toml, a wheel, a hash-locked install, pip-audit and pipx.

A tool that lives as a folder of scripts works on the author's machine. For a teammate or a server it needs a name, a version, a declared list of dependencies and a command to run. In this lesson you will turn hashcheck from the argparse lesson into a package: a src layout, a pyproject.toml, two commands installed as entry points, an editable install for development and a wheel for everyone else. Then you will lock every dependency to an exact version and a SHA-256 hash, watch a tampered download and a missing hash both stop the install, audit the lock for known vulnerabilities, and install the tool for end users with pipx.

A package: src layout and pyproject.toml

Unpack the lesson files (the box at the top of this page) in your home directory and work in ~/ps-package. They hold the project as this lesson starts it: the modules from the argparse lesson in src/hashcheck/ (hashcheck.py renamed to cli.py), a pyproject.toml, and that lesson's make_uploads.sh, which the first command below runs to build the sample uploads and the IOC list; a copy of the list goes into feed/, where a test server will publish it. src/hashcheck/ has an __init__.py (one docstring line), which makes it a package: modules that are imported together under one name. Inside a package, modules import each other relatively (from .cli_args import build_parser; the dot means "from this package"), which is the only change to their code, as grep shows. One new module gives the package a real dependency: hashcheck-fetch downloads an IOC list with requests, the library the HTTP lesson justified.

"""hashcheck-fetch: download an IOC list, refuse an unusable one, and replace the old file atomically.
 
Exit status: 0 saved; 2 usage error; 4 the download failed, or the list is unusable or cannot be
saved (the old file is kept).
"""
import argparse
import io
import os
import sys
import tempfile
from pathlib import Path
 
import requests
 
from .hashing import IocListError, load_iocs
 
MAX_BYTES = 10_000_000  # an IOC list larger than this is a mistake or an attack
 
 
def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(prog="hashcheck-fetch", allow_abbrev=False,
                                     description="Download an IOC list for hashcheck.")
    parser.add_argument("url", help="where the list is published (https://)")
    parser.add_argument("--out", required=True, type=Path, metavar="FILE", help="where to save it")
    parser.add_argument("--allow-http", action="store_true", help="accept http:// (a lab server)")
    args = parser.parse_args(argv)
    if not (args.url.startswith("https://") or args.allow_http):
        parser.error("the URL must start with https://: this list decides what counts as malware")
    try:
        text = download(args.url)
        iocs = load_iocs(io.StringIO(text), args.url)  # check before saving
        save(text, args.out)
    except (requests.RequestException, UnicodeDecodeError, IocListError, OSError) as err:
        print(f"hashcheck-fetch: {err}; {args.out} not changed", file=sys.stderr)
        return 4
    print(f"saved {len(iocs)} indicators to {args.out}")
    return 0
 
 
def download(url: str) -> str:
    with requests.get(url, timeout=(3.05, 10), stream=True) as response:  # connect, read
        response.raise_for_status()
        body = b""
        for chunk in response.iter_content(65536):
            body += chunk
            if len(body) > MAX_BYTES:
                raise IocListError(f"{url}: more than {MAX_BYTES} bytes")
    return body.decode("utf-8")  # UTF-8 whatever the server claims; bad bytes are an error
 
 
def save(text: str, out: Path) -> None:
    """Readers see the old list or the new one, never half of each (the files lesson)."""
    fd, tmp = tempfile.mkstemp(dir=out.parent, prefix=f".{out.name}.")
    try:
        with open(fd, "w", encoding="utf-8") as f:
            f.write(text)
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, out)
    except BaseException:
        os.unlink(tmp)  # never leave the temporary file behind
        raise

An IOC list decides what the scanner treats as malware, so the fetcher is strict about it. It accepts only https:// unless --allow-http says otherwise (for a local test server only), because anyone on the network path could replace a list fetched over plain HTTP with harmless hashes; signing the list itself is scripting-adv material. It reads the body in chunks up to a size cap, decodes it as UTF-8 itself (for a text/plain answer without a charset, requests would guess ISO-8859-1 and turn every non-ASCII label into garbage), and checks it with load_iocs() before saving. save() is the files lesson's pattern: a temporary file in the same directory, fsync(), then os.replace(), and the temporary file removed on any failure, so a failed or empty download never destroys a working list and a full disk is exit status 4, not a traceback. The package is described in pyproject.toml:

[build-system]
requires = ["hatchling==1.32.4"]  # the tool that turns this directory into a wheel
build-backend = "hatchling.build"
 
[project]
name = "hashcheck"
version = "0.1.0"
description = "Find files whose SHA-256 is on an IOC list, and quarantine them."
requires-python = ">=3.14"  # the version it is tested on
dependencies = [
    "requests>=2.33,<3",  # a range: the lock file pins the exact version
]
 
[project.scripts]
hashcheck = "hashcheck.cli:main"
hashcheck-fetch = "hashcheck.fetch:main"
 
[tool.hatch.build.targets.sdist]
include = ["/src"]  # without this, the sdist takes almost every file in the directory

[build-system] names the build backend, the tool that turns the source into an installable file. This project uses hatchling, pinned to one version so every build uses the same tool. [project] is the package's metadata. dependencies is a range, >=2.33,<3: a package that pinned requests==2.34.2 could never be installed next to another tool that needs a different 2.x, and every security fix in requests would need a new release of hashcheck. Exact versions belong in a lock file, later in this lesson. The floor 2.33 is not arbitrary; the audit section shows why. [project.scripts] defines entry points: the command hashcheck calls the function main in the module hashcheck.cli. The last table is explained with the build.

That failure is the point of the src layout. From the project directory, Python cannot import hashcheck by accident; tests and commands always use the installed package, so a broken install shows up on your machine, not a teammate's.

Editable install, entry points and a wheel

First a venv for development. Ubuntu's python3-venv puts pip 25.1.1 into a new venv, a year behind the current release, so the first step upgrades it to a pinned 26.2.1 and every platform in this lab behaves the same. The tool output in this lesson was captured without a terminal, as in a CI job, so you see no colours, spinners or progress bars; at a terminal you will see those.

pip install -e . (editable) reads pyproject.toml, installs the pinned hatchling into a temporary build environment, resolves the requests range to 2.34.2 and its four dependencies, and installs hashcheck so that it points at src/: edits to the source take effect without reinstalling. For each entry point pip writes a small launcher into the venv's bin/:

The launcher's first line is the venv's own Python, so the command works without activating the venv, including from cron. The import path shows the editable install reading src/. hashcheck-fetch refused the plain http:// URL with a usage error, saved the list from the local test server with --allow-http, and for the missing file printed the requests error, kept the old file and exited 4, its documented status. The last line stopped the test server. The build tools go into a second venv, because build, pip-tools and pip-audit are not needed at run time:

python -m build makes two files: an sdist (source distribution, a .tar.gz of the project source) and a wheel (a .whl zip that pip unpacks without running any build step). Here is the first attempt, with the [tool.hatch.build.targets.sdist] table removed:

A 6.4 MB sdist, holding the whole .tools venv (2043 files), the IOC list, the sample uploads and the fetched list. Without instructions, hatchling leaves out only what .gitignore excludes (this directory has none) and a few fixed names such as .venv, .git and __pycache__; everything else goes in. That is why the development venv, .venv, stayed out while the tools venv went in: its name is .tools. A token file or a .env sitting in the project would have been published the same way. The include = ["/src"] table limits the sdist to the source (hatchling adds pyproject.toml and its metadata itself). The real build:

The wheel is 8 KB: the six modules, METADATA with the name, version and the Requires-Dist range, entry_points.txt with the two commands, and RECORD, a list of every file with its hash. py3-none-any in the name means pure Python, any platform. That single file is what you hand to other people. The 5 KB sdist now holds only src/, pyproject.toml and PKG-INFO.

Lock the versions, with hashes

A range says what works; a lock file says exactly what was tested. pip-compile from pip-tools resolves the ranges once and writes every package, direct and indirect, with an exact version, and with --generate-hashes the SHA-256 of every file PyPI offers for it:

Five packages; # via says who needed each one. The header is pip-compile's own record of how to reproduce the file, so it can list options you did not type, such as --no-index. charset-normalizer has a hash for each of its platform wheels, which is why its entry runs to many lines (left out above). --pip-args "--timeout 60" gives pip 60 seconds per network read instead of its default 15; two lab runs failed on a slow PyPI download without it. Install it into a fresh venv, the way a server would:

--require-hashes makes pip compare the SHA-256 of every file it installs with the lock and refuse anything that does not match. The tool's own wheel is installed in a second command with --no-deps, so pip resolves nothing beyond the lock. To see the check fail, build a wheelhouse (a local directory of wheels, used for offline installs with --no-index --find-links) and change one byte of one wheel:

pip names the file, the two hashes the lock allows for idna==3.20 (its wheel and its sdist) and the hash it got: 64 hex digits each, as every SHA-256 is. It stopped before installing anything, including the three files that were fine; pip list shows only pip. Put the good wheel back and the same offline install succeeds:

Now a lock line without a hash, as happens when someone adds a package by hand:

Hash-checking mode fails closed: one requirement without a hash, and nothing is installed. pip helpfully prints the hash of the file it just downloaded. Pasting that line into the lock proves only that the file matches itself; it compares the download with nothing you trusted before. Add packages to pyproject.toml and run pip-compile again. A hash also cannot tell you whether the locked version was malicious when you locked it: it proves "unchanged since the lock", not "safe". The lock also covers only what the tool runs with. The tools that build and audit it (build, the hatchling inside its build environment, pip-tools, pip-audit) were pinned here but not hash-checked, and they produce the artifact and the verdict; in CI they deserve a lock of their own, which the scripting-adv lessons on packaging and the supply chain set up.

Audit the lock with pip-audit

pip-audit, maintained by the Python Packaging Authority, checks exact versions against published advisories; by default it asks PyPI's vulnerability API, so it needs the network. Point it at the lock, which contains the indirect dependencies too. --disable-pip audits the pinned lines as written instead of installing them into a temporary venv, which works because the lock is fully pinned and hashed:

Clean today. To see a finding, lock an old release of requests in a throwaway file:

# A throwaway input for the audit demo: an old release with published advisories.
requests==2.32.3

Exit status 1, which fails a CI step. The table says four, but it holds two advisories, each listed twice with the same ID and aliases: count the distinct IDs, not the rows. CVE-2024-47081 (GHSA-9hjg-9r4m-mvj7) can leak .netrc credentials through a crafted URL and is fixed in 2.32.4; CVE-2026-25645 (GHSA-gc5v-m9x4-r6x2) is insecure reuse of a temporary file and is fixed in 2.33.0. That second fix is why pyproject.toml says >=2.33. An audit only knows published advisories: a freshly hijacked release is malicious, not "known vulnerable", and passes. Run it on every change to the lock and on a schedule, because new advisories appear for versions you locked long ago.

pipx for the people who run it

Teammates who only want the command should not manage venvs. pipx installs each application into a venv of its own and puts only its commands on PATH. Ubuntu 26.04 packages pipx 1.8.0 (sudo apt install pipx), which is enough for install, run and upgrade; the lab installs 1.17.7, the current release when it ran, from PyPI into its own venv, so the version is pinned:

The commands went into ~/.local/bin, which did not exist when this shell started. Ubuntu's ~/.profile adds that directory to PATH at login if it exists, so the next login finds it (pipx ensurepath does the same for other setups):

The launcher points at pipx's own venv for hashcheck (-E makes Python ignore PYTHON* environment variables). pipx runpip hashcheck list shows how the dependencies got there: pipx resolved the requests range at install time. It did not read your lock, so the versions match only because nothing newer exists today. Where the exact, hash-checked set matters, on servers, install into a venv from the lock as above. pipx isolates dependencies, not privileges: the tool runs with your user's rights.

Where packages come from matters as much as which versions. --extra-index-url adds a second index, and pip then treats both indexes as equal and installs the highest version it finds. If your company's private index serves an internal package, say acme-ioc-utils, and someone uploads acme-ioc-utils 99.0 to public PyPI, the public one wins. That is dependency confusion. Use one index (a private mirror that proxies PyPI) with --index-url, and keep hashes in the lock so a swapped file fails the install.

Two newer lock formats are worth knowing. PEP 751 standardised pylock.toml, which pip 25.1 and later can write experimentally (pip lock) and pip 26.1 and later can install from, also experimentally (pip install -r pylock.toml), and uv, a fast installer written in Rust, keeps its own uv.lock. The scripting-adv lesson on shipping automation compares these locking strategies.

Try this

Give hashcheck a --version option that reads the installed version with importlib.metadata.version("hashcheck") (in cli_args.py: parser.add_argument("--version", action="version", version=...)), and raise the version in pyproject.toml to 0.2.0. Work in a copy of the project, build only the wheel (python -m build --wheel --outdir try-dist try-project), create a fresh venv, install the lock with --require-hashes and the new wheel with --no-deps, and check that hashcheck --version prints hashcheck 0.2.0. Then try the shortcut people reach for: python3 try-project/src/hashcheck/cli.py --version. It fails with ImportError: attempted relative import with no known parent package, because a file run directly is not part of a package; installed packages and entry points are how a package runs.

Takeaway

Declare dependency ranges in pyproject.toml, lock exact versions with hashes, and install servers only with --require-hashes from that lock, so that a changed or unlisted file stops the install. Audit the lock regularly, limit what the sdist contains, and remember that pipx isolates dependencies, not privileges.

Related