Resilience: exceptions, retries, timeouts & cleanup

Exception hygiene, retry/backoff, timeouts, and context managers that always clean up.

A good line cook keeps plating when a burner dies. They slide the pan to another ring, keep moving, and the order still lands on time. Your automation works the same kitchen. Networks blip. APIs (application programming interfaces, the services your code calls over the network) rate-limit you. Disks fill. A script that assumes every step succeeds is a cook who walks out the moment one burner fails. Resilient code expects trouble: it retries what is worth retrying, gives up fast on what is not, puts a clock on every wait, and cleans up after itself no matter how it exits. For a defender this is not cosmetic. A brittle security job that dies halfway can leave a firewall rule half-applied, a secret sitting in a temp file, or a lock held forever, and the worst ones fail while still reporting success.

Exception hygiene

Catching an exception is like putting a net under a tightrope. You want it to catch the falls you can do something about, not the stagehand walking past. The narrower the net, the better. A bare except: is a net with no holes. It catches the error you expected and also the ones you did not: a bug in your own code, the interpreter running out of memory, and the Ctrl-C an operator just pressed to stop a runaway job.

You pressed Ctrl-C to kill it, and the script shrugged and finished "cleanly". Swap except: for except Exception: and that Ctrl-C sails straight through, because KeyboardInterrupt is deliberately not an Exception (it descends from BaseException, one level up). So the rule is simple to state and hard to skip: catch the narrowest type you can actually handle, and let everything else propagate to the caller. When you do turn a low-level failure into a domain error your caller understands, keep the receipt. raise FetchError(...) from err chains the original cause onto the new one, so the traceback (the stack printout Python shows on an unhandled error) prints both the error you reported and the one that really happened underneath it.

# fetch.py: catch narrow, chain the cause, let the rest propagate
import sys
import httpx
 
class FetchError(Exception):
    """A fetch failed in a way the caller is expected to handle."""
 
def fetch(url: str) -> bytes:
    try:
        r = httpx.get(url, timeout=httpx.Timeout(10.0, connect=3.0))
        r.raise_for_status()
        return r.content
    except httpx.HTTPStatusError as e:
        # a 404 or 403 is not retryable; report it as our error, keep the cause
        raise FetchError(f"{url} -> {e.response.status_code}") from e
    except httpx.TimeoutException as e:
        raise FetchError(f"{url} timed out") from e
    # ConnectError, ReadError, etc. propagate unchanged: the caller decides
 
def main():
    body = fetch(sys.argv[1])
    print(len(body))
 
if __name__ == "__main__":
    main()

Read that output top to bottom. The real cause, a 404 from the server, sits above the line "The above exception was the direct cause of the following exception", and your domain error sits below it. Without the from err, you would see only the second half and spend an afternoon guessing where the 404 came from. The failure mode to fear most is the silent one. A handler that swallows an error, logs nothing, and returns as if all is well turns a failed control into a green check mark. If your key-rotation script eats the API error, your dashboard says the keys rotated. They did not.

Retries, backoff, and jitter

Retrying is knocking on a door. If nobody answers, you wait and knock again, and you wait a little longer each time. But if the door has a sign that says "closed, back next week" (a 404, a 401), knocking harder will never open it. So retry only when two things are true at once: the operation is safe to repeat, and the failure is actually temporary. Safe to repeat means idempotent (doing it twice leaves the same result as doing it once, like reading a file or setting a value to 5). Temporary means a timeout, a connection reset, or a 5xx (a 500-range server error), the kind of failure that might clear on its own.

A 400 (bad request) or a 401 (unauthorized) fails the same way every time, so retrying it burns time and, in your logs, looks exactly like someone brute-forcing a login. Back off exponentially, doubling the wait each round, so you do not pile onto a service that is already on the floor. Then add jitter (a small random amount mixed into each wait) so a whole fleet of machines does not retry on the same tick. Without jitter, a thousand clients that failed together wake together and hit the recovering service in one synchronized wave, the thundering herd, and knock it back down. Cap two things: how many attempts, and how long in total. tenacity (a Python retry library) lets you write all of that as a decorator instead of a hand-rolled loop.

# retry.py: retry only transient failures, capped exponential backoff + jitter
import sys
import httpx
from tenacity import (
    retry, stop_after_attempt, stop_after_delay,
    wait_exponential_jitter, retry_if_exception_type,
)
 
TRANSIENT = (httpx.TimeoutException, httpx.ConnectError)
 
def log_attempt(state):
    exc = state.outcome.exception()
    wait = state.next_action.sleep
    print(f"attempt {state.attempt_number} failed "
          f"({type(exc).__name__}); sleeping {wait:.2f}s", file=sys.stderr)
 
@retry(
    retry=retry_if_exception_type(TRANSIENT),
    wait=wait_exponential_jitter(initial=0.5, max=20),
    stop=(stop_after_attempt(5) | stop_after_delay(30)),
    before_sleep=log_attempt,
    reraise=True,
)
def get(client, url):
    r = client.get(url, timeout=httpx.Timeout(10.0, connect=3.0))
    if r.status_code >= 500:
        raise httpx.ConnectError("upstream 5xx, retryable")
    r.raise_for_status()      # 4xx is raised here and NOT retried
    return r
 
try:
    with httpx.Client() as client:
        get(client, sys.argv[1])
except TRANSIENT as e:
    print(f"gave up after retries: {type(e).__name__}", file=sys.stderr)
    sys.exit(1)

Four failures, four growing sleeps with a bit of random on top, then it gives up cleanly on the fifth. Look at stop_after_delay(30) sitting next to stop_after_attempt(5): whichever budget runs out first ends the retries, so a slow endpoint cannot stretch five attempts into ten minutes. And notice what never gets retried. The raise_for_status() on a 4xx throws a type that is not in TRANSIENT, so tenacity lets it through on the first try instead of hammering a request that will fail identically forever.

Bound every wait

A network read with no timeout is a phone call you never hang up. The other side went quiet, and you are still holding the receiver an hour later. Here is the part that surprises people: a plain Python socket (the low-level network connection) has no timeout at all by default. Not a long one. None. It will wait forever for bytes that may never come.

# slow.py: a socket with no timeout waits forever
import socket
s = socket.socket()
s.connect(("10.255.255.1", 80))   # blackholed address: connect blocks with no timeout to stop it
print("connected")

timeout 5 is GNU timeout, a small command that kills whatever it runs after five seconds; exit code 124 is its signature for "I had to kill this". That is your outer bound. But the real fix belongs in the code: set the wait explicitly. socket.setdefaulttimeout(3) caps every new socket, and an HTTP client gives you four separate dials in httpx.Timeout(10.0, connect=3.0) for connect (opening the connection), read (waiting for each chunk of the reply), write (sending your request body), and pool (waiting for a free connection from the pool). This is a live attack surface, not a tidiness issue. Point a worker at a host that accepts the connection and then dribbles one byte every thirty seconds forever, and a worker with no read timeout sits there pinned. Do it to every worker in the pool and the pool stops doing work. That low-and-slow trick has a name, slowloris, and an unbounded read hands it to an attacker for free.

Give yourself a second net at the operating-system level too. If the job runs under systemd (the service manager that starts and supervises programs on modern Linux), a couple of lines cap its whole lifetime and hand it a throwaway temp directory.

[Unit]
Description=Rotate API keys
After=network-online.target
Wants=network-online.target
 
[Service]
Type=oneshot
ExecStart=/opt/fetch/.venv/bin/python3 /opt/fetch/rotate.py
# hard ceiling for a oneshot job: SIGTERM at 120s, SIGKILL 30s later.
# (RuntimeMaxSec would be ignored here; TimeoutStartSec is the one that bites.)
TimeoutStartSec=120
TimeoutStopSec=30
# a private /tmp that is thrown away when the unit stops, even on a crash
PrivateTmp=yes

TimeoutStartSec=120 tells systemd to send SIGTERM (the polite "please stop" signal) at two minutes and, if the job ignores it, SIGKILL (the un-catchable "stop now" signal) thirty seconds later. Reach for the right directive here. On a Type=oneshot job, TimeoutStartSec is what bounds the run, while RuntimeMaxSec looks right but does nothing, because it only starts its clock once a service reaches the running state and a oneshot never gets there. systemctl show reads the ceiling back as TimeoutStartUSec=2min, which is how you confirm it actually took after an edit and a daemon-reload. Now no single hung request can keep this job alive past two minutes, whatever the code forgot to bound.

Cleanup that always runs

Locking up a shop for the night has to happen however the night ends, whether you leave on time or a pipe bursts and you run for the door. with is that lock-up routine. A resource you open has to be released on every path out, including the path where something raises, and with guarantees it. Files, locks, network sessions, temp directories, subprocesses: they either support with already or can be wrapped in it. For your own setup-and-teardown pairs, contextlib.contextmanager turns a single function into one, and ExitStack lets you stack several without nesting five with blocks or forgetting the last one.

# clean.py: teardown runs on every exit path, even a crash mid-work
from contextlib import contextmanager, ExitStack
import tempfile, shutil, os, sys
 
@contextmanager
def workdir(prefix="tool."):
    d = tempfile.mkdtemp(prefix=prefix)       # mode 0700, owned by you
    print(f"created {d}", file=sys.stderr)
    try:
        yield d                               # hand the resource to the body
    finally:
        shutil.rmtree(d, ignore_errors=True)  # ALWAYS runs
        print(f"removed {d}", file=sys.stderr)
 
with ExitStack() as stack:
    d = stack.enter_context(workdir())
    f = stack.enter_context(open(os.path.join(d, "secret.tmp"), "w"))
    f.write("token=deadbeef")
    raise RuntimeError("boom, halfway through the job")

The job blew up on line 19, and the temp directory still got removed on the way out, so the final ls finds nothing (exit 2 from the shell means the glob matched no files). mkdtemp gave you that directory with mode 0700 (readable and writable only by you), which is exactly what you want for a file that briefly holds a token. ExitStack unwinds in reverse: the file closes before the directory it lives in is deleted, the same order you would do it by hand.

Two habits keep this honest. Run grep -rn 'except:' yourpkg/ and make every bare handler either narrow down or justify itself in a comment, because a silent except is where failed controls go to hide. Then prove your cleanup by raising inside the with, as clean.py does, and watching the temp directory vanish. Learn its one limit while you are there: finally runs on exceptions, normal exits, and a caught SIGTERM, but a kill -9 (SIGKILL) cannot be caught, so anything it leaves behind is yours to sweep on the next start. For secrets that must never linger, let the platform help: PrivateTmp=yes gives the service its own /tmp that is destroyed when it stops, so even a hard kill cannot strand a token on disk.

Try this

Work through “Cleanup that always runs” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a retry loop around a write can double the damage. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related