Hashes, HMAC and password storage

File checksums, webhook HMAC verification, tokens and why sha256 is wrong for passwords.

Three questions come up constantly in security automation: are these the exact bytes that were published, did this message really come from the sender who shares my key, and how do I store a password so that a stolen copy is of little use? Python's standard library answers all three with hashlib, hmac and secrets. In this lesson you will hash files and check them against a SHA256SUMS list, verify signed webhook deliveries over their raw bytes with a replay window, generate tokens, and store passwords with scrypt, measuring on the lab machine why SHA-256 is the wrong tool for that last job.

Fingerprint a file with hashlib.file_digest

A cryptographic hash function turns any amount of data into a short fixed-size value, the digest. The same bytes always give the same digest, a change of a single bit gives an unrelated one, and no one knows how to construct two different inputs with the same SHA-256 digest (for MD5 and SHA-1 people do, which is why they are unfit for this). So a digest is a fingerprint: compare two, and you know whether the bytes are identical. hashlib.file_digest() (Python 3.11 and later) reads a file in chunks and feeds each one to the hash, so memory use stays small for a file of any size:

"""Print the SHA-256 of each file named on the command line, in the format sha256sum uses."""
import hashlib
import sys
 
for name in sys.argv[1:]:
    with open(name, "rb") as f:  # binary: a hash is computed over bytes, not text
        digest = hashlib.file_digest(f, "sha256")
    print(f"{digest.hexdigest()}  {name}")

Unpack the lesson files (the box at the top of this page) in your home directory and work in ~/ps-crypto. Besides the scripts, they contain release/ with two small archives and their SHA256SUMS list, and event.json, so your digests match the ones shown here:

The Python digests match sha256sum character for character, because SHA-256 is a standard. A hash is computed over bytes, so the file must be opened in binary mode ("rb"); file_digest() refuses a text-mode file with a ValueError. Before 3.11 the same job was a loop calling update() on 64 KiB chunks, which you will still see in older code; it gives the same result.

Check a download against SHA256SUMS

Projects publish a SHA256SUMS file next to their downloads: one line per file, the digest, two spaces and the name. The lesson's release/ directory has two archives and their list:

"""Check files against a SHA256SUMS list, as sha256sum -c does.
 
Exit status: 0 every file matches, 1 a file is missing or differs, 2 the list itself is unusable.
"""
import hashlib
import re
import sys
from pathlib import Path
 
# "<64 hex digits><space><space or *><file name>", as sha256sum writes it
LINE = re.compile(r"([0-9a-fA-F]{64}) [ *](.+)")
 
 
def parse(sums: Path) -> list[tuple[str, str]]:
    entries = []
    for number, line in enumerate(sums.read_text(encoding="utf-8").splitlines(), 1):
        match = LINE.fullmatch(line)
        if not match or "/" in match[2] or match[2] in (".", ".."):  # plain names only, no paths
            raise ValueError(f"{sums}:{number}: not a checksum line: {line!r}")
        entries.append((match[1].lower(), match[2]))
    if not entries:
        raise ValueError(f"{sums}: no checksums at all")  # an empty list must not mean "all good"
    return entries
 
 
def main(argv: list[str]) -> int:
    if len(argv) != 1:
        print("usage: verify_sums.py SHA256SUMS", file=sys.stderr)
        return 2
    sums = Path(argv[0])
    try:
        entries = parse(sums)
    except (OSError, ValueError) as err:
        print(f"error: {err}", file=sys.stderr)
        return 2
    failed = 0
    for expected, name in entries:
        try:
            with open(sums.parent / name, "rb") as f:
                actual = hashlib.file_digest(f, "sha256").hexdigest()
        except OSError as err:
            print(f"{name}: MISSING ({err.strerror})")
            failed += 1
            continue
        print(f"{name}: {'OK' if actual == expected else 'FAILED'}")
        failed += actual != expected  # True adds 1, False adds 0
    return 1 if failed else 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The regular expression accepts the two forms sha256sum writes, HASH name and HASH *name (the * marks binary mode, which makes no difference on Linux). Hex digits are compared in lower case, so an upper-case list still matches. The list is untrusted input too: a line that is not a checksum, a name with a path in it, or an empty list is exit status 2, because "no checksums, therefore nothing failed" must never read as success.

Both files match, and sha256sum -c (from the Bash capstone) agrees; it reads the names relative to the current directory, so it runs in a subshell that changes into release and leaves your shell where it was. Now change one bit of one archive in a copy of the directory. In the python3 -c line, bytearray is a changeable copy of the file's bytes, and b[100] ^= 1 flips the lowest bit of byte 100 (^ is exclusive or):

cmp confirms the files differ in a single byte, and both tools report FAILED with exit status 1. The remaining failures:

A missing file fails like a mismatch. The empty list and the list naming ../../etc/passwd are refused before anything is hashed. One limit applies to all of this: a checksum proves the file matches the list, not that the list is genuine. An attacker who can replace the download can usually replace a SHA256SUMS next to it. Fetch the list over a trusted channel and check its signature; the scripting-adv lesson on release gates covers signing and verification.

HMAC: prove who sent a webhook

A webhook is an HTTP request another service sends you when something happens: an alert fired, a scan finished. Anyone who knows your URL can send one, so the sender signs it with a key you both hold. An HMAC (hash-based message authentication code) is a hash computed over the message together with that shared key: without the key nobody can produce a matching value, so a valid HMAC shows the message is unchanged and came from a key holder. Services differ in exactly what they sign. GitHub puts sha256= plus the HMAC of the raw body in X-Hub-Signature-256; Slack signs v0:timestamp:body and tells receivers to reject timestamps more than five minutes from their own clock. The lab uses the second pattern, with a timestamp header:

"""Sign and verify webhook deliveries: HMAC-SHA256 over the timestamp, a dot and the raw body."""
import hashlib
import hmac
import time
 
WINDOW = 300  # seconds; an older (or future-dated) delivery is refused
 
 
class Rejected(Exception):
    """The delivery is not authentic, or not fresh."""
 
 
def sign(secret: bytes, timestamp: str, body: bytes) -> str:
    mac = hmac.new(secret, timestamp.encode("ascii") + b"." + body, hashlib.sha256)
    return "sha256=" + mac.hexdigest()
 
 
def verify(secret: bytes, body: bytes, timestamp: str, signature: str) -> None:
    """Return quietly for a good delivery; raise Rejected otherwise."""
    if not (timestamp.isascii() and timestamp.isdigit()):
        raise Rejected("malformed timestamp")
    if not signature.isascii():  # compare_digest refuses non-ASCII str with a TypeError
        raise Rejected("malformed signature")
    expected = sign(secret, timestamp, body).encode("ascii")
    if not hmac.compare_digest(expected, signature.encode("ascii")):  # bytes, constant time
        raise Rejected("signature does not match")
    if abs(time.time() - int(timestamp)) > WINDOW:  # checked after the MAC: the timestamp is signed
        raise Rejected(f"timestamp outside the {WINDOW} s window")

hmac.new(key, message, hashlib.sha256) computes the HMAC. hmac.compare_digest() compares in constant time: == stops at the first differing character, so its running time leaks how much of a forged signature was right, and enough measurements can rebuild a valid one. The comparison is done on bytes: with two str values, compare_digest() raises TypeError for any non-ASCII character, which an attacker controls in a header. The timestamp is checked after the signature because it is part of what was signed, so it cannot be changed without breaking the MAC. isascii() and isdigit() together, because isdigit() alone accepts digits such as ². send.py plays the sender and receive.py the receiver; both read the shared key from webhook-secret, a 0600 file that the first command below creates, and send.py signs event.json from the lesson files:

"""The sender's side: sign BODY_FILE with the shared secret and store it as a delivery.
 
Usage: send.py BODY_FILE OUT_DIR [AGE_SECONDS]   (AGE_SECONDS backdates the timestamp)
"""
import sys
import time
from pathlib import Path
 
from webhook import sign
 
body = Path(sys.argv[1]).read_bytes()
age = int(sys.argv[3]) if len(sys.argv) > 3 else 0
timestamp = str(int(time.time()) - age)
secret = Path("webhook-secret").read_bytes().strip()
out = Path(sys.argv[2])
out.mkdir(exist_ok=True)
(out / "body").write_bytes(body)
(out / "headers").write_text(
    f"X-Signature-Timestamp: {timestamp}\nX-Signature-256: {sign(secret, timestamp, body)}\n",
    encoding="utf-8",
)
print(f"{out}: {len(body)} bytes signed, timestamp {age} s in the past")

"""Check a stored webhook delivery: DIR/body holds the raw bytes, DIR/headers the headers.
 
Exit status: 0 accepted, 1 rejected, 2 usage error or unreadable delivery.
"""
import json
import sys
from pathlib import Path
 
from webhook import Rejected, verify
 
 
def read_headers(path: Path) -> dict[str, str]:
    headers = {}
    for line in path.read_text(encoding="utf-8").splitlines():
        name, _, value = line.partition(":")
        headers[name.strip().lower()] = value.strip()
    return headers
 
 
def main(argv: list[str]) -> int:
    if len(argv) != 1:
        print("usage: receive.py DELIVERY_DIR", file=sys.stderr)
        return 2
    delivery = Path(argv[0])
    try:
        secret = Path("webhook-secret").read_bytes().strip()
        body = (delivery / "body").read_bytes()  # the bytes exactly as they arrived
        headers = read_headers(delivery / "headers")
    except OSError as err:
        print(f"error: {err}", file=sys.stderr)
        return 2
    try:
        verify(secret, body, headers.get("x-signature-timestamp", ""), headers.get("x-signature-256", ""))
    except Rejected as err:
        print(f"rejected: {err}")
        return 1
    event = json.loads(body)  # parse only after the signature checked out
    print(f"accepted: {event['type']} {event['id']} for {event['host']}")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The genuine delivery was accepted, and changing "high" to "low" broke the signature. The third case is the one that catches real receivers: d3/body holds the same JSON data, parsed and written back with json.dumps(), which adds a space after every colon and comma and drops the final newline. Equivalent to a person, different bytes to a hash function, so it was rejected. Web frameworks that hand you parsed JSON make this mistake easy; always take the raw request body for the check.

A correctly signed delivery with a timestamp ten minutes old was refused: a captured request cannot be replayed next week. Inside the window, though, the same delivery was accepted a second time. The window limits replays to five minutes; to stop them entirely, remember the IDs you have processed, which is the Try this exercise. Last, a signature header with a non-ASCII character:

Comparing str values raised TypeError, which in a web service becomes a crash or a 500 error for every such request; verify() rejects the header cleanly instead.

Tokens: use secrets, not random

Webhook keys, API tokens and password-reset links need values nobody can predict. The random module is built for simulations: its generator is fully determined by its internal state, which can be reconstructed from enough outputs, and its documentation says not to use it for security. The secrets module draws from the operating system's cryptographic generator:

token_urlsafe(32) encodes 32 random bytes as 43 characters that are safe in URLs and headers; token_hex(16) gives 32 hex digits. The values differ on every run. 32 bytes (256 bits) is a sensible size for keys and tokens.

Passwords: slow, salted, and not SHA-256

A hash cannot be reversed, but it can be guessed: hash candidate inputs until one matches. That only fails when there are too many candidates. A common mistake in log pipelines is to "anonymise" IP addresses or user names by hashing them, and those have very few candidates:

"""Why a SHA-256 of an IP address is not anonymous: there are few enough candidates to try them all."""
import hashlib
import ipaddress
import sys
 
wanted = sys.argv[1]
for guesses, ip in enumerate(ipaddress.ip_network("203.0.113.0/24"), 1):
    if hashlib.sha256(str(ip).encode()).hexdigest() == wanted:
        print(f"{ip} (found after {guesses} guesses)")
        break
else:
    print("not in 203.0.113.0/24")

78 guesses. The else belongs to the for: it runs only when the loop finished without break, here when no address matched. Passwords are in the same position, because people choose from a small space and attackers work through lists of common ones. What matters is how many guesses per second the hash allows. Measured on the lab machine, on one core:

"""Measure hashes per second on this machine: fast SHA-256 against deliberately slow scrypt."""
import hashlib
import os
import time
 
 
def per_second(work, seconds: float = 1.0) -> float:
    count, start = 0, time.perf_counter()
    while time.perf_counter() - start < seconds:
        work()
        count += 1
    return count / (time.perf_counter() - start)
 
 
salt = os.urandom(16)
guess = b"correct horse battery staple"
print(f"sha256:                 {per_second(lambda: hashlib.sha256(salt + guess).digest()):12,.0f} per second")
print(f"scrypt n=2**17 r=8 p=1: {per_second(lambda: hashlib.scrypt(guess, salt=salt, n=2**17, r=8, p=1, maxmem=2**28)):12,.1f} per second")

About 3 million SHA-256 hashes per second against about 4 scrypt hashes, a factor of roughly 700,000 (the numbers vary between runs and machines; these are from one core of the lab VM while other labs were running). At that SHA-256 rate all 4.3 billion IPv4 addresses take under half an hour on one core, and a list of ten million common passwords a few seconds; graphics cards are much faster again at SHA-256. A salt, a random value stored next to each hash, makes identical passwords hash differently and forces the attacker to guess each account separately; it does nothing about speed. Password storage needs a function that is slow and needs a lot of memory on purpose, so that guessing is expensive on any hardware. OWASP's first choice is Argon2id (the argon2-cffi package); the standard library offers hashlib.scrypt, and OWASP's minimum for it is n=2**17, r=8, p=1, which uses 128 MiB per hash. Try that directly:

OpenSSL, which does the work, refuses to use more than 32 MiB unless you raise the limit with maxmem. passwords.py sets it, and stores the parameters with each hash so they can be raised later:

"""Store and check passwords with scrypt from the standard library (no dependency needed)."""
import base64
import hashlib
import hmac
import secrets
 
N, R, P = 2**17, 8, 1  # OWASP's minimum for scrypt: 128 MiB of memory per hash
MAXMEM = 2**28  # 256 MiB; without it OpenSSL refuses anything above 32 MiB
 
 
def b64(data: bytes) -> str:
    return base64.b64encode(data).decode("ascii")
 
 
def hash_password(password: str) -> str:
    salt = secrets.token_bytes(16)  # a new random salt for every stored password
    key = hashlib.scrypt(password.encode(), salt=salt, n=N, r=R, p=P, maxmem=MAXMEM, dklen=32)
    return f"scrypt${N}${R}${P}${b64(salt)}${b64(key)}"  # parameters stored with the hash
 
 
def verify_password(password: str, stored: str) -> bool:
    try:
        name, n, r, p, salt, key = stored.split("$")
        n, r, p = int(n), int(r), int(p)
        salt, key = base64.b64decode(salt), base64.b64decode(key)
    except ValueError:
        return False
    if name != "scrypt":
        return False
    candidate = hashlib.scrypt(password.encode(), salt=salt, n=n, r=r, p=p, maxmem=MAXMEM, dklen=len(key))
    return hmac.compare_digest(candidate, key)

"""Hash one password twice and check a right and a wrong guess."""
import time
 
from passwords import hash_password, verify_password
 
start = time.perf_counter()
stored = hash_password("correct horse battery staple")
print(f"hashing took {time.perf_counter() - start:.2f} s")
print(f"stored:  {stored}")
print(f"again:   {hash_password('correct horse battery staple')}")
print("right password:", verify_password("correct horse battery staple", stored))
print("wrong password:", verify_password("Correct horse battery staple", stored))

One hash took about a fifth of a second, acceptable at login and ruinous for an attacker. The same password stored twice gives two different strings, because each has its own salt. The right password verifies and a change of one letter's case does not; compare_digest() compares the derived keys in constant time. Peak memory was 144 MiB, the 128 MiB scrypt needed plus the interpreter: budget for it on a busy login service. Both Python builds on the lab machine (Ubuntu 3.14.4 and upstream 3.14.7, both on OpenSSL 3.5.5) gave the same results.

Try this

Add replay protection to the receiver. Write receive_once.py DIR that verifies a delivery as receive.py does, then reads the event's id from the body (inside the signed data, so it cannot be changed without breaking the HMAC) and rejects it with rejected: replayed delivery evt-0042 and exit status 1 if the ID is already in a file seen-ids; otherwise it appends the ID and prints accepted: evt-0042. Verify with a fresh delivery: python3 send.py event.json d6, then run receive_once.py d6 twice. The first run is accepted, the second rejected. Then confirm what you fixed: receive.py d6 still accepts it any number of times within the five minutes.

Takeaway

Hash files with hashlib.file_digest and treat checksum lists as untrusted input; verify HMACs over the raw bytes with hmac.compare_digest on bytes, inside a time window; make tokens with secrets; and store passwords only with a slow, salted, memory-hard function such as scrypt or Argon2id, never a plain or salted SHA-256.

Related