Calling APIs safely

requests, auth, timeouts, retries.

Calling an API is like sending a courier into an office building you don't run. You show a badge at the front desk, you wait on the building's schedule, and you trust whatever envelope comes back out. An API (application programming interface, a service other programs talk to over the network) works the same way. You send an HTTP request (Hypertext Transfer Protocol, the language browsers and servers use to pass messages) to a web address called an endpoint, and the service answers with structured data, almost always JSON (JavaScript Object Notation, text laid out as labeled key/value pairs). Security automation runs on these couriers: checking an IP address (Internet Protocol address, the number that identifies a machine on a network) against a threat feed, pulling alerts out of your EDR (endpoint detection and response, the agent watching each laptop and server), opening a ticket. Every one of those calls crosses a network you don't own, carries a secret you must not leak, and lands on a service that can stall, lie, or throttle you.

Four ways a naive call fails

A plain requests.get(url) in a SOC (security operations center, the team watching alerts around the clock) script breaks in four predictable ways. It hangs: the requests library ships with no default timeout, so a single unresponsive API can freeze your whole enrichment run until someone notices the dashboard went cold. It leaks: an API key pasted straight into source code rides along into git history and, eventually, into somebody else's hands. It storms: a while-loop that keeps hammering a failing endpoint gets your key rate-limited or banned, which blinds every other tool sharing that key. And it swallows poison: turn off certificate checking, or never look at the response body, and an attacker sitting on the network path can feed your automation invented intelligence that it will act on without blinking. A hardened client is a client with a written-down answer to each of those four.

The building blocks

Before the code, four ideas. A session is a reusable client object, like keeping one courier on staff instead of hiring a stranger for every errand. It holds TCP connections open between calls, which is called connection pooling (TCP, Transmission Control Protocol, the protocol that opens a reliable link with a handshake before any data moves). Reusing the connection skips redoing the TLS handshake on every request. TLS (Transport Layer Security, the encryption that puts the S in HTTPS, meaning HTTP carried over that encrypted channel) is the lock on the line; your client checks the server's certificate against a CA bundle (certificate authority bundle, a file listing the issuers your machine trusts), which requests ships through the certifi package. A retry policy is the rule for which failures earn a second try and how long to wait first. And the secret, your API key, comes from an environment variable, a .env file loader, or a secrets manager, so it never sits in the code itself.

A client you can trust

Here is a full client for AbuseIPDB, a public service that scores how often an IP address has been reported for abuse. Read it once, then take it apart.

import os
import sys
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
 
API = "https://api.abuseipdb.com/api/v2/check"
 
def build_session() -> requests.Session:
    retry = Retry(
        total=5,
        backoff_factor=1,          # 1st retry immediate, then ~2s, 4s, 8s, 16s (urllib3 2.x)
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"],   # never auto-retry writes
        respect_retry_after_header=True,
    )
    s = requests.Session()
    s.mount("https://", HTTPAdapter(max_retries=retry))
    s.headers["Key"] = os.environ["ABUSEIPDB_API_KEY"]   # KeyError = fail fast
    s.headers["Accept"] = "application/json"
    return s
 
def check_ip(session: requests.Session, ip: str) -> dict:
    r = session.get(
        API,
        params={"ipAddress": ip, "maxAgeInDays": 90},
        timeout=(5, 15),           # (connect, read) seconds
    )
    r.raise_for_status()           # 4xx/5xx -> exception, not silent garbage
    return r.json()["data"]
 
if __name__ == "__main__":
    data = check_ip(build_session(), sys.argv[1])
    print(f"{data['ipAddress']} score={data['abuseConfidenceScore']}"
          f" reports={data['totalReports']}")

The layering is the point. requests hands the actual connection work to a lower library called urllib3, and the Retry object lives down there. So it counts attempts, sleeps between them, and obeys the server's Retry-After header before your code ever sees a response. allowed_methods=["GET"] limits automatic retries to idempotent requests, ones you can safely repeat because they don't change anything on the server (asking 'what is the score for this IP' five times is harmless; opening a ticket five times is not). The timeout tuple splits connect (how long to wait for the TCP and TLS handshake to finish) from read (how long to wait for each chunk of the answer), because an API that never picks up and one that picks up then goes silent are different problems with different fixes. raise_for_status() turns HTTP error codes, the three-digit status codes like 403 Forbidden or 500 Internal Server Error, into a Python exception your caller has to handle, instead of a dict full of nonsense that flows downstream and blows up somewhere far from the cause.

Run it, then break it on purpose

That is the happy path: one call, one score, verification on, timeout armed. Now prove the timeout, the longest you are willing to wait, actually protects you. Call an endpoint that stalls for ten seconds while you allow only three.

Three seconds, a clean exception, and the run steps to the next indicator instead of hanging until a human notices. The retry policy covers the opposite case, a fast failure that might clear on its own, using exponential backoff (each wait roughly double the one before). With backoff_factor=1 the first retry fires immediately, then the waits stretch to about 2, 4, 8, and 16 seconds, so five transient 503 Service Unavailable errors cost tens of seconds of patience rather than five instant hammer-blows. When the server answers a 429 Too Many Requests with a header that reads Retry-After: 30, urllib3 sleeps for that exact thirty seconds in place of its own backoff, which is the difference between cooperating with a rate limit and picking a fight with it.

The verify=False trap

When a corporate TLS-inspecting proxy breaks certificate checks, the top search result tells you to pass verify=False. Watch what that actually does against a server whose certificate your machine does not trust.

What breaks at scale

A rate limit is the server's cap on how many requests your key may make in a window, a minute or a day; cross it and you get 429 Too Many Requests. At small scale you never meet one. At scale, enriching ten thousand IP addresses from last night's alerts, you hit it in the first minute, and if five scripts share one org-wide key, a single greedy script starves the rest. Budget the calls: batch when the API supports it, cache answers locally so you never ask the same question twice, and read a 429 as 'slow down', not 'retry harder'. Pagination (an API handing back a large result set one page at a time, using a cursor or a page number) needs a hard cap on loop iterations, because a server bug that returns the same cursor forever turns your poller into an infinite loop that eats the same page all night. And never log request headers at DEBUG level. That is exactly how API keys leak into your own log pipeline, sitting in plaintext for everyone with SIEM (security information and event management, the platform your logs get shipped to) access to read.

Where this design stops

Know the edges. urllib3's Retry is deliberately blunt about calls that change state: if a POST (an HTTP request that creates or updates something on the server) that opens a ticket times out, you cannot tell whether the ticket was created, so an automatic retry risks a duplicate. Handle those by hand, or send an idempotency key when the API offers one (a unique token the server uses to recognize and drop a repeat). For async work or HTTP/2 (the newer version of HTTP that runs many requests over one connection at once), reach for httpx, a modern client with the same ideas: explicit timeouts, a Client object, verification on by default. And treat every response as untrusted input. r.json() proves the body parsed, not that it holds the shape you expect. Check the fields you depend on, because a loud KeyError at the call site beats a None quietly riding into a blocking decision, and reach for pydantic (a library that checks data against a declared schema and rejects what does not fit) once the structure gets complicated.

One enriched IP is a lookup. Ten thousand of them, pulled from a single night of alerts, is a data-processing job with its own failure shapes: memory that balloons, JSON that arrives as a gigabyte stream, a parser that chokes on one malformed line and takes the batch down with it. That is the next lesson, where the data you now know how to fetch safely meets the tools that chew through it without running the box out of memory. Before you scale anything up, run that grep for verify=False across every repository that touches an API key, and turn the ones you find back on.

Related