Hashing & integrity: hashlib, hmac

Fingerprint files and verify without leaks.

Every file on your laptop, every release you download, every webhook that hits your server is a run of bytes. You constantly need to answer two questions about those bytes: did they arrive exactly as sent, and did they really come from who they claim? Hashing answers both, and Python hands you the tools in two small standard-library modules, hashlib and hmac.

A fingerprint for a pile of bytes

A fingerprint is tiny next to the whole person, yet no two people share one. A hash does the same trick for data. Feed any bytes through a hash function (a one-way math routine that turns any input into a fixed-size string called a digest) and you get back the same short length every time, whether the input was a one-line note or a huge disk image. Change a single byte of the input and the digest changes completely, in a way nobody can steer. The one-way part is what makes it useful: going from file to digest takes a blink, and going from digest back to the original file is not possible. That drives the two jobs in this lesson. hashlib computes digests. hmac adds a secret key so a digest also proves who produced it. Both ship with Python, so there is nothing to install.

Fingerprint a file without eating your RAM

The obvious approach reads the whole file into memory and hashes it. That is fine for a config file and a disaster for a 4 GB (gigabyte) image, because you would pull all 4 GB into RAM (random-access memory, your computer's fast working space) at once. On a build agent with little memory to spare, the process dies. The fix uses a handy property of hashes: you can build one up piece by piece, the way a cashier scans a cart one item at a time and still reaches the same total as ringing everything together. Updating the hash with chunk A and then chunk B gives the exact same digest as updating it once with A and B glued together. So you read a 64 KB (kilobyte) slice, fold it in, drop it, and read the next slice. Memory stays flat no matter how large the file grows.

import hashlib, sys
 
def sha256_of(path, chunk_size=65536):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(chunk_size):
            h.update(chunk)
    return h.hexdigest()
 
if __name__ == "__main__":
    for path in sys.argv[1:]:
        print(f"{sha256_of(path)}  {path}")

The `while chunk := f.read(chunk_size)` line grabs 64 KB at a time and stops when `read` returns empty bytes at the end of the file. Opening with `"rb"` gives raw bytes, which is what a hash wants, never decoded text. Run coreutils' own `sha256sum` on the same file and you get the identical string, character for character. That is the point of a standard algorithm: everyone computes the same digest for the same bytes.

Verify a download against a published checksum

A tamper-evident seal on a medicine bottle tells you at a glance whether the contents were opened. A published checksum is that seal for a download. The project ships the file and also publishes a small SHA256SUMS file listing the digest it expects. You hash your copy and compare. A match means the bytes on your disk are exactly the bytes the project built. A mismatch means a truncated download, a corrupted mirror, or someone who swapped the file for their own.

8aa1146a0f996d43b372674327198f633348f20fb4431c78e8f6eece378f5161  app-2.4.1.tar.gz

import sys
from checksum import sha256_of
 
expected, path = sys.argv[1], sys.argv[2]
actual = sha256_of(path)
 
if actual == expected:
    print(f"OK: {path} matches the published checksum")
else:
    print(f"FAIL: {path} does NOT match")
    print(f"  expected: {expected}")
    print(f"  actual:   {actual}")
    sys.exit(1)

The exit code matters in a pipeline. verify.py returns 0 on a match and 1 on a mismatch, so a CI (continuous integration, the automated build-and-test step that runs on every commit) job fails loudly instead of shipping a bad artifact. One honest limit sits underneath all of this: a checksum only helps if you fetched it from somewhere you trust, over HTTPS (HyperText Transfer Protocol Secure, an encrypted web connection) from the project's own site, not from the same mirror as the file. An attacker who can replace the download can replace a checksum sitting next to it. Hashes catch accidents and lazy tampering; cryptographic signatures (GPG, short for GNU Privacy Guard) close the gap against an attacker who controls the mirror. Treat the checksum as your first layer, not your only one.

HMAC: a fingerprint only the keyholder can forge

A plain hash has one gap: anyone can compute it, so it proves nothing about who made it. You and a friend agree on a secret knock. Anyone can bang on your door, but only someone who knows the pattern taps out the right rhythm, so the knock itself tells you who is there. HMAC (Hash-based Message Authentication Code, a hash mixed with a shared secret key) does the same for data. Here is the gap it closes: if a webhook (an automated HTTP call one service sends another when an event happens) attached a plain sha256 of its body, an attacker could invent a fake event, attach a matching hash, and you could not tell the forgery from the real thing. With HMAC, you and the sender agree on a secret ahead of time. The sender hashes the message together with the secret; you repeat the calculation and check that the two line up. Without the secret, nobody can produce the right digest, so a valid HMAC proves two things at once: the message was not altered, and it came from someone holding the key. GitHub signs every webhook this way. Set a webhook secret and each delivery arrives with a header, X-Hub-Signature-256: sha256=<hmac>, that your endpoint recomputes and checks.

import hashlib, hmac
 
# The secret you configured in GitHub -> Webhooks. In real code, read it from an
# environment variable or a secrets manager, never hard-code it.
SECRET = b"s3cr3t-webhook-key"
 
# The exact raw request body GitHub sent (bytes, not the parsed JSON object).
payload = b'{"action":"opened","number":42}'
 
# What GitHub puts in the header, computed on its side with the same secret.
received_header = "sha256=" + hmac.new(SECRET, payload, hashlib.sha256).hexdigest()
 
def is_valid(secret, body, header):
    if not header.startswith("sha256="):
        return False
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    sent = header.split("=", 1)[1]
    return hmac.compare_digest(expected, sent)  # constant-time, not ==
 
print("X-Hub-Signature-256:", received_header)
print("valid payload   ->", is_valid(SECRET, payload, received_header))
 
forged = b'{"action":"opened","number":999}'
print("forged payload  ->", is_valid(SECRET, forged, received_header))
 
wrong_key = "sha256=" + hmac.new(b"guessed-key", payload, hashlib.sha256).hexdigest()
print("wrong secret    ->", is_valid(SECRET, payload, wrong_key))

One detail decides whether this works: hash the raw bytes of the request body, exactly as they arrived, before any JSON (JavaScript Object Notation, a common text format for structured data) parsing. If you parse the body and then re-serialize it, key order or spacing can shift, the bytes change, and your HMAC will not match even for a genuine request. GitHub signed the precise bytes on the wire, so those are the bytes you feed to hmac.new.

Compare secrets in constant time

Notice the code compares with hmac.compare_digest, not ==. A cheap combination lock that clicked a little louder as each correct digit dropped into place would be easy to crack: you would listen your way through one digit at a time. A normal string compare has the same leak. == stops at the first byte that differs, so a forged signature whose first byte happens to be right takes a hair longer to reject than one that is wrong immediately. An attacker who can measure those tiny differences across many requests can rebuild the correct signature byte by byte. That is a timing attack. compare_digest checks the full length every time, so its runtime says nothing about how close a guess was. Use it for every secret comparison you make: webhook signatures, session tokens, API (application programming interface) keys.

Three ways to get hashing wrong

First, md5 and sha1 are broken for security. Researchers can craft two different inputs that produce the same md5 or sha1 digest, a collision, which means the digest no longer stands uniquely for its input. Here is how easy all three are to reach in Python:

md5 and sha1 are fine for a non-security job like a cache key or spotting an accidental duplicate file. For anything an attacker might target, such as verifying a download or signing a webhook, use sha256 or stronger and never the older two.

Second, plain hashlib is wrong for passwords. sha256 is built to be fast, and a modern GPU (graphics processing unit, a chip that runs enormous numbers of calculations in parallel) can try billions of sha256 guesses a second. That speed is the enemy: if your user database leaks, it lets the attacker brute-force every password quickly. Password storage needs a slow, salted, purpose-built function: bcrypt or argon2 (installed with pip install bcrypt or argon2-cffi). They are deliberately slow and mix a unique random salt (extra random bytes) into each password, so two identical passwords store as different digests. If you truly cannot add a dependency, hashlib's own pbkdf2_hmac and scrypt are acceptable fallbacks, but reach for argon2 first. Third, always hash the raw bytes you received, never a parsed-and-rebuilt copy, or a genuine signature check will fail on you.

Related