Command-line tools with argparse

Arguments, subcommands, stdin, an exit-code contract and an honest dry-run.

A script that someone else will run, from cron, a CI job or an incident runbook, needs a command line that says what it accepts, refuses bad input before doing anything, and reports its result in a way another program can act on. In this lesson you will build hashcheck, a tool that finds files whose SHA-256 is on an IOC list (indicators of compromise, here known-bad file hashes) and can move them into quarantine. You will use argparse for options, types, choices and subcommands, read standard input when a file name is -, turn off option abbreviations, define an exit-code contract, and write a --dry-run that really changes nothing and a quarantine step that never overwrites evidence.

Arguments, types and choices

Unpack the lesson files (the box at the top of this page) in your home directory and work in ~/ps-cli. The tool needs something to scan: make_uploads.sh builds a small uploads/ tree with two different harmless samples that are both called invoice.pdf, an identical copy of one of them and a 2 KB report, and an iocs.txt list with one SHA-256 and a label per line:

#!/usr/bin/env bash
# Builds the lesson's uploads/ tree and iocs.txt from scratch. The samples are harmless text;
# iocs.txt lists the SHA-256 of two of them, as a feed of known-bad file hashes would.
set -euo pipefail
rm -rf uploads
mkdir -p uploads/a uploads/b uploads/c uploads/mail
printf 'lab sample A: harmless text standing in for a malicious invoice\n' > uploads/a/invoice.pdf
printf 'lab sample B: a different harmless file with the same name\n' > uploads/b/invoice.pdf
cp uploads/a/invoice.pdf uploads/mail/invoice.pdf  # an identical copy of sample A
{
  printf 'Quarterly report, lab fixture.\n\n'
  for n in $(seq -w 1 30); do
    printf '%s. Host web%s patched, no findings, reviewed by the on-call engineer.\n' "$n" "$n"
  done
} > uploads/c/report.txt  # 2162 bytes: larger than --max-size 2K
 
digest() { sha256sum < "$1" | cut -d ' ' -f 1; }
{
  printf '# lab IOC list: SHA-256, then a label (test values, not real malware hashes)\n'
  printf '%s  dropper-2026-091\n' "$(digest uploads/a/invoice.pdf)"
  printf '%s  invoice-lure-2026-114\n' "$(digest uploads/b/invoice.pdf)"
  printf '%s  loader-2026-077\n' 10ecafc91e767032812b8456977674656ebe074a191e76c8e3502ccd58799d11
} > iocs.txt
wc -c uploads/*/* iocs.txt

The tool is split into four short modules: hashing.py and cli_args.py in this section, the main module hashcheck.py in full in the section on exit codes, and quarantine.py in the last section. hashing.py reads the IOC list and hashes the inputs with hashlib.file_digest, which you met in the hashing lesson:

"""Reading the IOC list and hashing the inputs, for hashcheck."""
import hashlib
import re
import stat
import sys
from pathlib import Path
from typing import IO
 
# One indicator per line: 64 hex digits, then an optional label. '#' starts a comment.
IOC_LINE = re.compile(r"(?P<sha256>[0-9a-fA-F]{64})(?:\s+(?P<label>\S.*))?")
STDIN = Path("-")
 
 
class IocListError(Exception):
    """The IOC list cannot be used: a malformed line, or no indicators at all."""
 
 
def load_iocs(stream: IO[str], name: str) -> dict[str, str]:
    """Return {sha256: label}. Every line must be an indicator, a comment or empty."""
    iocs = {}
    for number, line in enumerate(stream, 1):
        line = line.split("#", 1)[0].strip()
        if not line:
            continue
        match = IOC_LINE.fullmatch(line)
        if not match:
            raise IocListError(f"{name}, line {number}: not a SHA-256 indicator: {line[:40]!r}")
        iocs[match["sha256"].lower()] = match["label"] or "unlabelled"
    if not iocs:  # an empty feed must never turn into "nothing matched"
        raise IocListError(f"{name}: no indicators")
    return iocs
 
 
def read_iocs(name: str) -> dict[str, str]:
    if name == "-":
        return load_iocs(sys.stdin, "stdin")
    with open(name, encoding="utf-8") as stream:
        return load_iocs(stream, name)
 
 
def digest(path: Path, max_size: int) -> str:
    """SHA-256 of a regular file, or of stdin for '-'. OSError and ValueError mean: not checked."""
    if path == STDIN:
        return hashlib.file_digest(sys.stdin.buffer, "sha256").hexdigest()
    info = path.lstat()  # the entry itself: a symbolic link is not followed
    if not stat.S_ISREG(info.st_mode):
        raise ValueError(f"{path}: not a regular file (a link, directory or device), not checked")
    if info.st_size > max_size:
        raise ValueError(f"{path}: {info.st_size} bytes is over --max-size, not checked")
    with path.open("rb") as stream:
        return hashlib.file_digest(stream, "sha256").hexdigest()

load_iocs() refuses a malformed line and an empty list with IocListError. That second rule matters more than it looks: if a feed download comes back empty and the tool quietly checks against nothing, every run reports "no match". IO[str] in its type hint means "an open text stream", a file or sys.stdin. digest() looks at the entry itself with lstat() and checks only a regular file: a symbolic link, a directory or a device is refused with ValueError, as is a file over the size limit, and OSError passes through for a file it cannot open; the caller decides what those mean. sys.stdin.buffer is standard input as bytes, which file_digest() needs. The command line lives in cli_args.py:

"""The command line of hashcheck: commands, options, types and choices."""
import argparse
from pathlib import Path
 
UNITS = {"K": 2**10, "M": 2**20, "G": 2**30}
 
 
def size(text: str) -> int:
    """A --max-size value such as 500, 64K or 10M, as a number of bytes."""
    multiplier = UNITS.get(text[-1:].upper(), 1)
    digits = text[:-1] if multiplier > 1 else text
    if not (digits.isascii() and digits.isdigit()):
        raise argparse.ArgumentTypeError(f"invalid size {text!r}: use a number with an optional K, M or G")
    return int(digits) * multiplier
 
 
def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="hashcheck",
        description="Find files whose SHA-256 is on an IOC list.",
        epilog="exit status: 0 no match, 1 match found, 2 usage error, 3 incomplete run, 70 internal error",
        allow_abbrev=False,
    )
    # Options every command takes, defined once and passed to each command as a parent.
    common = argparse.ArgumentParser(add_help=False)
    common.add_argument("--iocs", required=True, metavar="FILE",
                        help="IOC list: one SHA-256 and a label per line ('-' reads stdin)")
    common.add_argument("--max-size", type=size, default="100M", metavar="SIZE",
                        help="skip larger files and report the run as incomplete (default: %(default)s)")
    common.add_argument("paths", nargs="+", type=Path, metavar="PATH",
                        help="files to check ('-' reads stdin)")
 
    commands = parser.add_subparsers(dest="command", required=True, metavar="COMMAND")
    # allow_abbrev is not inherited: every subcommand parser needs it too.
    scan = commands.add_parser("scan", parents=[common], allow_abbrev=False,
                               help="print the files that match")
    scan.add_argument("--format", choices=["text", "json"], default="text",
                      help="output format (default: %(default)s)")
    move = commands.add_parser("quarantine", parents=[common], allow_abbrev=False,
                               help="move the files that match into a quarantine directory")
    move.add_argument("--into", required=True, type=Path, metavar="DIR", help="quarantine directory")
    move.add_argument("--dry-run", action="store_true", help="print what would be moved, change nothing")
    return parser

argparse.ArgumentParser collects the arguments a program accepts; each add_argument() call adds one. A name starting with -- is an option, anything else is a positional argument. nargs="+" means one or more values, metavar is the name shown in the help, and help is its description; %(default)s inserts the real default, so the help cannot drift from the code. type= is any function that turns the text into a value: Path makes a pathlib.Path, and size() turns 10M into 10485760. When a type function raises argparse.ArgumentTypeError, argparse prints its message as a usage error. choices= accepts only the listed values. required=True on an option is allowed but unusual; here it keeps the IOC list explicit, so no run can fall back to a list nobody chose. The parents and add_subparsers parts are the next section.

The help comes from the declarations. The scan found three matching files (two of them identical copies of the same sample) and exited 1, which the next sections turn into a contract. Now the input argparse refuses:

Every refusal has the same shape: the usage line of the command that failed, then PROG: error: MESSAGE, on stderr, and exit status 2, before a single file was opened. The --max-size message is the one size() wrote. One detail differs between Python builds. Ubuntu's 3.14.4 prints the choices bare, (choose from text, json); upstream CPython 3.14.5 put the quotes back (CPython issue gh-130750), so the lab's 3.14.7 prints:

Do not match the exact error text in scripts; match the exit status. Python 3.14 also colours help and usage on a colour terminal (NO_COLOR=1 turns it off). The lab terminal reports itself as TERM=dumb, so the output here is plain:

Two more 3.14 details. ArgumentParser(suggest_on_error=True) adds "maybe you meant 'json'?" to a wrong choice or command; it is off by default in 3.14. And argparse.FileType, which opened files while parsing, is deprecated in 3.14 ("Simply open files after parsing arguments", says its warning): an argument that is a file is parsed as a name or a Path, and the program opens it later, where it can handle the error.

Subcommands, and the abbreviation trap

hashcheck has two verbs, scan and quarantine, the way git has commit and push. add_subparsers(dest="command", required=True) creates them: each add_parser() returns a parser of its own, and the chosen name ends up in args.command. The options both commands take are declared once on a helper parser, common, created with add_help=False so it does not bring a second -h, and passed to each command with parents=[common]:

By default argparse accepts any unambiguous prefix of a long option: --dry means --dry-run. For a tool that moves files, that is a risk in scheduled jobs. A prefix that is unique today can become ambiguous when a later version adds an option (the job starts failing), or it can match a new option spelled exactly like it, which wins over any prefix (the job silently changes meaning). allow_abbrev=False turns prefixes off. The catch, shown with a small parser:

"""allow_abbrev=False on the top-level parser only: what a subcommand still accepts."""
import argparse
 
parser = argparse.ArgumentParser(prog="abbrev", allow_abbrev=False)
parser.add_argument("--verbose", action="store_true")
commands = parser.add_subparsers(dest="command", required=True)
move = commands.add_parser("move")  # allow_abbrev not passed: the default, True
move.add_argument("--dry-run", action="store_true")
print(parser.parse_args())

The top-level parser rejected --verb, but the move subcommand still took --dry as --dry-run: allow_abbrev is not inherited by the parsers add_parser() creates. That is why cli_args.py passes allow_abbrev=False to every add_parser() call, and hashcheck refuses --dry. The error is reported by the top-level hashcheck parser, because a subcommand hands arguments it does not know back up to its parent.

An exit-code contract: stdout for data, stderr for problems

An exit-code contract is a documented promise about what each status means, so that cron, CI and wrapper scripts can act on it without parsing text. The Bash lesson on options set up the same convention: 0 for success, 1 for findings, 2 for usage errors, and any other code documented. In this course 2 means "the command could not run as asked": a usage error, and also input, configuration or setup the operator must fix before anything can be checked, such as an unusable IOC list, the way grep uses 2 for a file it cannot read. One more code is needed in Python, because of what an unhandled exception does:

Python ends with status 1 on any uncaught exception. In this contract 1 means "a match was found, and every input was checked", so a crash halfway through would read as a complete run with findings. hashcheck.py therefore catches every unexpected Exception in main() and returns 70 (EX_SOFTWARE in sysexits.h, "internal software error"), and documents it in the docstring and in the help epilog:

"""hashcheck: find files whose SHA-256 is on an IOC list, and optionally quarantine them.
 
Exit status: 0 no match; 1 at least one match; 2 usage error, unusable IOC list or quarantine
directory; 3 incomplete: a file was not read, was too large or was not moved (wins over 1);
70 internal error: the program failed in a way it does not expect (the run is incomplete).
"""
import json
import logging
import sys
 
from cli_args import build_parser
from hashing import STDIN, IocListError, digest, read_iocs
from quarantine import QuarantineError, destination, prepare, quarantine
 
OK, FOUND, USAGE, INCOMPLETE, INTERNAL = 0, 1, 2, 3, 70
log = logging.getLogger("hashcheck")
 
 
def main(argv: list[str] | None = None) -> int:
    logging.basicConfig(format="hashcheck: %(message)s")  # stderr, WARNING and above
    parser = build_parser()
    args = parser.parse_args(argv)  # a usage error exits 2 here, with a message on stderr
    if args.iocs == "-" and STDIN in args.paths:
        parser.error("--iocs - and a PATH of - cannot both read stdin")
    if args.command == "quarantine" and STDIN in args.paths:
        parser.error("quarantine needs files; stdin cannot be moved")
    try:
        return check(args)
    except Exception:  # a bug, or an input nobody planned for: it must never look like a match
        log.exception("internal error; the run is incomplete")
        return INTERNAL
 
 
def check(args) -> int:
    try:
        iocs = read_iocs(args.iocs)
        if args.command == "quarantine":
            prepare(args.into, create=not args.dry_run)
    except (OSError, UnicodeDecodeError, IocListError, QuarantineError) as err:
        log.error("cannot start: %s", err)
        return USAGE
    found = incomplete = False
    for path in args.paths:
        try:
            sha256 = digest(path, args.max_size)
        except OSError as err:  # missing, no permission
            log.warning("%s: %s", path, err.strerror)
            incomplete = True
            continue
        except ValueError as err:  # over --max-size, or not a regular file
            log.warning("%s", err)
            incomplete = True
            continue
        if sha256 not in iocs:
            continue
        found = True
        if args.command == "scan" and args.format == "json":
            print(json.dumps({"path": str(path), "sha256": sha256, "ioc": iocs[sha256]}))
        elif args.command == "scan":
            print(f"{path}  {iocs[sha256]}")
        elif args.dry_run:  # the plan, from the same code path; nothing below runs
            print(f"would move {path} -> {destination(args.into, path, sha256)}")
        else:
            try:
                print(f"moved {path} -> {quarantine(path, sha256, args.into)}")
            except (OSError, QuarantineError) as err:  # FileExistsError included: never overwrite
                log.warning("%s: not moved: %s", path, err)
                incomplete = True
    return INCOMPLETE if incomplete else FOUND if found else OK
 
 
if __name__ == "__main__":
    sys.exit(main())

main(argv=None) takes the argument list as a parameter; parse_args(None) reads sys.argv[1:], so the program behaves normally and the testing lesson can call main(["scan", ...]) directly and check the returned status. sys.exit(main()) turns the return value into the exit status. parser.error() is for rules argparse cannot express, such as "not stdin twice": it prints the usage and exits 2 like any other usage error. The work itself is in check(), so that main() can wrap all of it in one try; log.exception() logs the message with the traceback, which is what the person debugging the failure needs. Results go to stdout with print(); problems go to stderr through logging, configured once in main() as the errors lesson recommends. To see the internal-error path, replace digest with a function that divides by zero, a stand-in for a real bug:

Two warnings reached the terminal while the three JSON lines went into matches.ndjson: stderr was not redirected. Exit status 3 says the run was incomplete although it found matches; a CI job that treats anything but 0 as a failure stops either way, and one that alerts on 1 learns that it must look at the warnings. A symbolic link counts as not checked too, because following it could hash (and later move) a file outside the tree you meant:

Standard input works both ways round:

A PATH of - hashes whatever arrives on stdin, as sha256sum - does, and --iocs - reads the list from a pipe. The third command is the empty-feed case: grep found nothing, and instead of "no match, exit 0" the tool refused the list with exit status 2. Both at once cannot work, since there is only one stdin, and parser.error() says so. A malformed line is reported with its line number.

An honest --dry-run, and a quarantine that never overwrites

Moving files is where a tool can destroy the evidence it was built to keep. Here is the first draft people often write, labelled unsafe:

"""UNSAFE first draft: moves each file into quarantine-naive/ under its own name."""
import sys
from pathlib import Path
 
into = Path("quarantine-naive")
into.mkdir(exist_ok=True)
for name in sys.argv[1:]:
    path = Path(name)
    path.rename(into / path.name)  # silently replaces a file that is already there
    print(f"moved {path} -> {into / path.name}")

Two different samples share the name invoice.pdf. The second rename() replaced the first without a word: on Linux, renaming onto an existing file replaces it (the os.rename documentation says so), and sample A is gone. quarantine.py avoids each part of that:

"""Move a file into a quarantine directory, never over an existing file, with a manifest line."""
import datetime
import hashlib
import json
import os
import stat
from pathlib import Path
 
 
class QuarantineError(Exception):
    """The quarantine directory is not private, or a file changed while it was being moved."""
 
 
def prepare(into: Path, create: bool) -> None:
    """Accept the directory only if it is a real directory, ours, and closed to everyone else."""
    if create:
        into.mkdir(mode=0o700, exist_ok=True)
    try:
        info = into.lstat()  # lstat: a symbolic link is judged as a link, not as its target
    except FileNotFoundError:
        return  # a dry run: the real run creates it
    if not stat.S_ISDIR(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o077:
        raise QuarantineError(f"{into}: must be a directory owned by uid {os.getuid()}, mode 0700")
 
 
def destination(into: Path, path: Path, sha256: str) -> Path:
    # The digest prefix keeps two different files that are both called invoice.pdf apart.
    return into / f"{sha256[:16]}-{path.name}"
 
 
def quarantine(path: Path, sha256: str, into: Path) -> Path:
    original = path.lstat()  # owner, mode and time of the original, for the manifest
    dest = destination(into, path, sha256)
    # O_EXCL: raise FileExistsError if dest exists, instead of replacing it as rename() would.
    fd = os.open(dest, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    try:
        with open(fd, "wb") as out, path.open("rb") as src:
            copied = hashlib.sha256()
            while chunk := src.read(2**20):  # a copy also works across file systems
                copied.update(chunk)  # hash the bytes that are stored, not an earlier read
                out.write(chunk)
            out.flush()
            os.fsync(out.fileno())  # the copy is on disk before the original is deleted
        if copied.hexdigest() != sha256:
            raise QuarantineError(f"{path} changed after it was hashed")
    except BaseException:
        dest.unlink()  # never leave half a copy, or a copy of other bytes, behind
        raise
    path.unlink()
    entry = {
        "time": datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds"),
        "original": str(path.absolute()),
        "sha256": sha256,
        "stored_as": dest.name,
        "uid": original.st_uid,
        "mode": stat.filemode(original.st_mode),
        "mtime": datetime.datetime.fromtimestamp(original.st_mtime, datetime.UTC).isoformat(),
    }
    with open(into / "manifest.ndjson", "a", encoding="utf-8") as manifest:
        manifest.write(json.dumps(entry) + "\n")
    return dest

The stored name starts with 16 hex digits of the file's hash, so different files with the same name no longer collide. os.open() with O_CREAT | O_EXCL creates the destination only if it does not exist and raises FileExistsError otherwise, in a single step, so there is no gap between "check" and "create" for another process to use. The copy goes into that new 0600 file inside a 0700 directory; a copy also works when the quarantine is on another file system, where rename() fails. while chunk := src.read(2**20): reads one mebibyte at a time: := assigns the chunk and tests it in one step, and an empty chunk (end of file) ends the loop. Each chunk is hashed as it is written, so the digest describes the bytes that were stored, not an earlier read; if the file changed after it was matched, the copy is removed and the original stays. os.fsync() puts the copy on disk before the original is deleted, as in the files lesson. Each move appends a JSON line to manifest.ndjson with the time, original path and hash, and the owner, mode and modification time the original had, which an investigator will ask for. prepare() runs once before the first move: mkdir(mode=0o700) sets the mode only when it creates the directory, so an existing directory, or a link to one, is accepted only if it is a real directory that belongs to you and is closed to everyone else. The Bash lesson "Command-line options" set the rule for a dry run: the same selection as the real run, no change, and a before-and-after comparison to prove it. In Python it has three parts: it goes through the same code as the real run, it stops before the first change, and it exits with the status the real run would report. Check the second claim the same way, not just the output:

The plan names every move, the listing of uploads/ is byte for byte what it was, and not even the quarantine directory was created. The plan also shows a problem before it happens: two moves go to the same destination, because uploads/mail/invoice.pdf is an identical copy of sample A. A quarantine directory that other users can write to is refused before anything moves, because they could read or replace the evidence:

The real run:

Both samples are stored under different names, with modes 0700 and 0600. The identical copy was refused (File exists), left where it was, and reported on stderr, and the exit status is 3 because the run did not do everything it was asked to. Running the same command again is safe:

The files already moved are no longer in uploads/, so the glob no longer names them; the copy is still refused, and nothing in quarantine was touched.

Try this

Write verify_quarantine.py DIR with argparse (allow_abbrev=False, DIR parsed as a Path) that reads DIR/manifest.ndjson, hashes each stored file and prints OK, CHANGED or MISSING with the file name. Its contract: 0 when every file is present and unchanged, 1 when any file is missing or changed, 2 for a usage error or an unreadable manifest (message on stderr). After the real run above it prints two OK lines and exits 0. Then break the evidence: append a line to quarantine/15c5924b1290721b-invoice.pdf and delete quarantine/165c2fe462be37c0-invoice.pdf; expect MISSING and CHANGED and exit status 1. Finally verify_quarantine.py nowhere must exit 2 with "cannot read the manifest", and no argument at all must give argparse's usage error, also 2.

Takeaway

Let argparse reject bad input before any work starts, pass allow_abbrev=False to every parser including subcommands, and document an exit-code contract in which an incomplete run can never look like a clean one. A --dry-run is only honest if it shares the real code path, stops before the first change and returns the real status; any move of evidence must refuse to overwrite.

Related