Secure Python: deserialization, validation & secrets

pickle/yaml.load dangers, input validation, and the secrets module.

Some files are a note you read. Others are a note that reads itself and does whatever it says. Python ships tools for both, and in code the two look almost identical. That gap matters more for security work than for almost anything else, because a security script runs with access most programs never get. It reads secrets, it calls internal services, and sometimes it runs as root (the all-powerful administrator account on Linux) on the very machine you are trying to protect. Hand an attacker a foothold there and you have handed them the keys to the building. Three habits close the three holes that keep coming back: never let untrusted data choose which code runs, check every input at the door, and use the right source of randomness when something has to be unguessable.

Loading Data Can Execute Code

Serializing means flattening a live object from memory into a string of bytes you can save to disk or send across the network. Deserializing is the reverse: rebuilding the object from those bytes. Python's built-in tool for this is pickle, and here is its sharp edge. Pickle does not only copy data. It can call functions to rebuild an object, and the bytes themselves say which functions to call. Think of shipping a flat-pack wardrobe with a small robot sealed inside the box. Open the box and the robot follows the printed instructions and assembles the furniture. Nothing stops those instructions from saying 'walk to the kitchen and turn on the stove' instead.

An object controls its own rebuild through a method named __reduce__, which returns the function to call and the arguments to hand it. An attacker writes a class whose __reduce__ returns os.system and a shell command. Unpickle their bytes and that command runs during the load, before you have read a single field of the result. This is remote code execution (RCE, an attacker running commands of their choosing on your machine), and it is one of the most common critical bugs filed against Python code.

That file looks like a harmless cached config object, a few dozen bytes on disk. Now a service loads it the way a hundred tutorials show you to load a cache.

No error. No warning. The load reported success, and partway through it the attacker's id command ran as the deploy user, who happens to sit in the docker group (which on most hosts is root in all but name, since you can mount the whole host filesystem from a container). YAML has the same trap. Its full loader honours tags like !!python/object/apply, which are the flat-pack robot wearing a different hat.

The fix is a reader that understands data and nothing else. yaml.safe_load builds only plain scalars, lists, and dictionaries, and it refuses the dangerous tags outright.

Silencing that machinery is not the same as fixing it. Since PyYAML 5.1 (released in 2019), calling yaml.load(text) with no loader was deprecated behind a printed warning. Modern PyYAML (6.0 and later) goes further and makes the Loader argument mandatory, so the call raises a TypeError until you pass one. The tempting way to quiet either of those is Loader=yaml.FullLoader or Loader=yaml.Loader. Do not. FullLoader has shipped real remote-code-execution bypasses (CVE-2020-1747 and CVE-2020-14343, entries in the public Common Vulnerabilities and Exposures catalogue, the shared registry of known security flaws), and Loader never pretended to be safe in the first place. yaml.safe_load is the only safe answer. Grep the codebase for 'yaml.load(' and treat every hit as a finding to fix.

So the rules stay short. For anything you did not create and protect yourself, never call pickle.load or pickle.loads, and always reach for yaml.safe_load. When you control both ends and want a plain data format, use JSON (JavaScript Object Notation, a text format that can only produce strings, numbers, lists, dictionaries, booleans, and null), because it has no way to name a function, let alone call one. If you genuinely must pass pickled objects between your own services, sign them: compute an HMAC (hash-based message authentication code, a fingerprint keyed by a secret both sides share) over the bytes and verify it before you ever call load, so a tampered payload is rejected without being deserialized.

Check Every Input at the Door

A nightclub does not check IDs at the bar. It checks them once, at the door, and everyone inside is already known to be old enough. Do the same with your data. The trust boundary is the line where information crosses from the outside world into your program: command-line arguments, environment variables, file contents, HTTP responses (replies coming back from a web server), queue messages. Everything crossing that line is a stranger until you have checked its type, its range, its length, and its format. Validate once at the edge, reject early with a clear error, and the core of your program only ever touches data that is already known-good.

pydantic makes that check declarative. You describe the shape you expect as a class, and parsing the input against it either hands you a clean typed object or raises an error. Here a job description arrives as JSON from an API (application programming interface, the agreed contract two programs use to talk to each other), and its count field must land between 1 and 1000.

Bad shape, bad type, or bad value, and you hear about it right at the boundary with a precise message, not three functions deep once the number is already driving a loop. One kind of input needs its own guard: file paths. A path from outside is a hotel key that should open exactly one room. Path traversal is a guest who scratches ../../manager-office onto the key and walks in where they do not belong. The value ../../../etc/passwd climbs out of your intended folder and into the system password file. Defend against it by resolving the path to its real absolute location and confirming it still sits under your base directory.

resolve() collapses every .. before you make a decision, and is_relative_to (available since Python 3.9) is the gate that actually matters. Do not build the check by hand with string prefixes. That is how a path like /srv/uploads-evil slips straight past a naive base.startswith test.

Use the Right Dice for Secrets

The random module is a card trick. The deck looks shuffled, but its order was fixed the instant it was seeded, and if you watch enough cards you can name the rest. Under the hood it is a Mersenne Twister, a PRNG (pseudo-random number generator, an algorithm that spits out a stream of values that only look random). Feed a standard cracking tool 624 consecutive outputs and it can rebuild the generator's entire internal state, then predict every value it will ever produce. Wonderful for shuffling test data. A disaster for a password-reset link.

Anything a person must never be able to guess has to come from the secrets module or from os.urandom. That covers tokens, API keys, session identifiers, password-reset links, and the nonces and salts (small one-off random values that stop repeated operations from looking identical) that cryptography leans on. Both draw from the operating system's CSPRNG (cryptographically secure pseudo-random number generator), the same unpredictable pool that TLS (Transport Layer Security, the encryption that protects web traffic) pulls its keys from. With a CSPRNG, past output tells an attacker nothing about the next byte.

token_urlsafe(32) gives you 32 bytes of real entropy encoded as a 43-character URL-safe string (a URL is a web address, and URL-safe means it is safe to paste straight into one), ready to drop into a link. token_hex(16) gives 16 bytes as 32 hex characters, handy for a key or an identifier. One more habit rides along with them. When you compare a secret a caller sent against the real one, do not use ==. A normal comparison stops at the first byte that differs, so the time it takes leaks how much of the secret the attacker has already guessed, one byte at a time. secrets.compare_digest takes the same amount of time no matter which bytes differ.

Use compare_digest for tokens, HMACs, and API keys. It is not a password hasher. Stored user passwords belong in a slow KDF (key derivation function, a one-way hash made deliberately expensive to run) such as Argon2 or bcrypt, which the argon2-cffi and bcrypt libraries give you so you never hand-roll the primitive yourself.

Let a Scanner Catch What You Miss

You will not catch every one of these in review, least of all in a pull request that touches forty files. bandit is a static analyzer (a tool that reads your code without running it) built for exactly this job. It knows the dangerous patterns by name and flags each one with a severity, a confidence, and a CWE (Common Weakness Enumeration, the industry's shared catalogue of weakness types). Point it at a file that has all three mistakes in it.

import pickle
import subprocess
 
 
def load_config(path):
    with open(path, "rb") as f:
        return pickle.load(f)
 
 
def run(cmd):
    return subprocess.call(cmd, shell=True)

Three findings, three severities. The Low one is the mere import of pickle. The Medium is the actual pickle.load. The High is shell=True, which bandit rates hottest because it is the shortest path from user input to a running shell. When a finding is new to you, follow its CWE link. It explains the whole class of bug rather than this single line, and that is usually where the real understanding clicks into place.

Wire the scan into CI (continuous integration, the automated pipeline that builds and checks every change as it lands) as a required gate. Run bandit -r . --severity-level medium --confidence-level medium, and treat a fresh B301 (pickle load), B506 (yaml.load), or B602 (shell=True) as a merge blocker. Fix the line, run the scan again, and watch the High count fall to zero. That last step is the whole point. You do not trust the diff, you trust the green scan.

Try this

Work through “Let a Scanner Catch What You Miss” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: shell=True turns an argument into a second command. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related