Recipe: NGINX as a reverse proxy

TLS edge, static files, proxying, and re-resolving names after a backend moves.

A backend container restarts during a deploy, comes back healthy, and every request through the proxy now answers 502 Bad Gateway until somebody reloads NGINX. The NGINX error log has the reason: connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://172.18.0.2:8000/". That address no longer belongs to the backend. This recipe builds the usual edge for a Compose stack (NGINX terminating TLS, serving static files and proxying /api/ to a private backend), reproduces that 502, and fixes it with Docker's DNS server. It runs on the main lab VM, secopslog-docker, in ~/lab/r-nginx with the lesson files.

The stack

Two services. web is nginxinc/nginx-unprivileged, the NGINX image built to run as UID 101 and to listen on 8080 and 8443. api is a twenty-line Python server that echoes what the proxy sent it, so you can see the headers arrive:

name: lab-edge
 
services:
  web:
    image: nginxinc/nginx-unprivileged:1.30-alpine@sha256:15c994d10d6d78658721c3bcafff14cb281fba2a4bdf9d5ba92c416a472516e3
    ports:
      - "127.0.0.1:80:8080"
      - "127.0.0.1:443:8443"
    volumes:
      - ./${EDGE_CONF:-default.conf}:/etc/nginx/conf.d/default.conf:ro
      - ./site:/usr/share/nginx/html:ro
    secrets: [tls_cert, tls_key]
    read_only: true
    tmpfs:
      - /tmp
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/healthz"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    depends_on:
      api:
        condition: service_healthy
    restart: unless-stopped
 
  api:
    image: python:3.14-slim@sha256:f85c5697265c178cc6887276c55fe16cf3d14ca35c3df6a5eab3b360534a55d2
    command: ["python", "/app/api.py"]
    init: true
    user: "65534:65534"
    volumes:
      - ./api/api.py:/app/api.py:ro
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 5s
    restart: unless-stopped
 
secrets:
  tls_cert:
    file: ./certs/fullchain.pem
  tls_key:
    file: ./certs/privkey.pem

Only web publishes ports, and only on the VM's loopback address; api has no ports: at all, so the one way to it is through the proxy on the Compose network. Both images are pinned by tag and digest. Both containers run with a read-only root filesystem, every capability dropped and no-new-privileges. NGINX only needs /tmp writable: the unprivileged image keeps its PID file and temporary request bodies there, hence the tmpfs. init: true on api puts a small init process in front of Python so docker compose stop takes a fraction of a second instead of the ten-second kill timeout ("CMD, ENTRYPOINT and PID 1" (Docker for beginners) explains why). The certificate and key arrive as Compose secrets. Both services have health checks; web checks only that NGINX answers its own /healthz, so a dead backend does not mark the proxy unhealthy. That is a choice: if you would rather have the orchestrator replace the proxy when the backend is gone, probe through it instead.

The NGINX configuration:

# Docker's embedded DNS server. Re-check answers every 10 s instead of
# trusting the record's 600 s TTL.
resolver 127.0.0.11 valid=10s ipv6=off;
resolver_timeout 3s;
 
server {
    listen 8080;
    location = /healthz { access_log off; return 200 "ok\n"; }
    location / { return 301 https://$host$request_uri; }
}
 
server {
    listen 8443 ssl;
    ssl_certificate     /run/secrets/tls_cert;
    ssl_certificate_key /run/secrets/tls_key;
 
    location / {
        root /usr/share/nginx/html;
        try_files $uri $uri/ /index.html;
    }
 
    location /api/ {
        # A variable in proxy_pass makes NGINX resolve "api" at request time.
        set $api_upstream http://api:8000;
        # With a variable, proxy_pass sends the URI unchanged, so strip /api here.
        rewrite ^/api/(.*)$ /$1 break;
        proxy_pass $api_upstream;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The plain-HTTP server answers /healthz and redirects everything else to HTTPS. The TLS server serves files from /usr/share/nginx/html and forwards /api/ to the backend with four headers: the original Host, the client address in X-Real-IP and X-Forwarded-For, and X-Forwarded-Proto: https, so the application knows the client used TLS even though the hop inside Docker is plain HTTP. The resolver line, the variable and the rewrite are the fix for the 502; the section after next explains them. ssl_protocols is not set, because the default since NGINX 1.23.4 is already TLS 1.2 and 1.3.

The key nobody can read

Create a self-signed certificate for localhost (a real deployment uses one from your CA or Let's Encrypt; the configuration is the same):

OpenSSL writes the private key as 0600, owned by ubuntu. Start the stack:

--wait waits for the health checks and exits 1 because web never became healthy. Outside Swarm mode, Compose hands a secret to the container as a read-only bind mount of the host file, so it keeps the host owner (UID 1000) and mode (0600). NGINX runs as UID 101, which cannot open it, and exits at startup. The fix is to make the file readable inside the container and protect it on the host with the directory instead:

Mode 0644 on the key looks wrong until you look at the directory: 0700 means no other host user can reach the file, while inside the container the bind mount exposes the file alone. Inside the container, though, 0644 lets every process read the key, whatever its user. If you can change the file's group, chgrp 101 (the image's nginx group) with mode 0640 narrows that to the NGINX user; the lab keeps 0644 because it is the setting that works without root on the host. The long-syntax uid, gid and mode fields on a Compose secret look like the cleaner fix, and they are ignored outside Swarm; "Recipe: WordPress and MySQL with secrets" shows the warning Compose prints. The ps output confirms the topology: api has no published port.

Checking each path

Plain HTTP gets a 301 to HTTPS. The static page comes off disk over TLS, and --cacert makes curl verify the self-signed certificate instead of skipping verification with -k. The API reply is the backend describing the request it received: the path arrived as /orders?page=2, without the /api prefix and with the query string intact. host is still localhost, and x_forwarded_proto is https. x_forwarded_for is 172.18.0.1, the network's gateway address: curl connected to a port published on 127.0.0.1, which Docker serves through docker-proxy, the userland proxy, and the proxy opens its own connection to NGINX from the gateway. A remote client reaching a port published on all addresses is DNATed and keeps its own address, unless the userland proxy handles it too ("Publishing ports and the packet path" covers both paths). Addresses on the Compose network vary between machines and runs.

One more claim worth checking, because it is repeated widely: that only root can bind ports below 1024, so non-root NGINX must use 8080:

Rootful Docker sets net.ipv4.ip_unprivileged_port_start=0 in every container's network namespace, so UID 101 could bind port 80 here. The unprivileged image listens on 8080 by convention, and that convention also works under rootless Docker and in Kubernetes with restricted policies. Run NGINX as non-root because it is non-root, not because of the port. "Capabilities, cap-drop and no-new-privileges" (Advanced container security) covers the sysctl.

Why the proxy keeps a dead address

Most NGINX examples write proxy_pass http://api:8000/;. static-upstream.conf is that version. The Compose file mounts whichever file EDGE_CONF names, so start web with it:

Now give the backend a new address. Recreating a container usually gets the same address back, because the old one is released first. It changes when something else takes the address while the backend is down: a one-off job, another service scaling up, a rolling deploy. Simulate that with a job container:

api is healthy at 172.18.0.4, and the proxy answers 502. With a literal hostname, NGINX resolves api once, when it loads the configuration, and keeps that address until the next reload. It is still sending requests to 172.18.0.2, which now belongs to the job container. Here the job does not listen on 8000, so the connection is refused. If it did, NGINX would send your API traffic, with its headers and cookies, to whatever container holds the old address. Docker's DNS record for api was correct the whole time.

The fix in default.conf has three parts. resolver 127.0.0.11 points NGINX at Docker's embedded DNS server, which every container on a user-defined network can reach at that address ("Networks, drivers and DNS"). valid=10s caps how long NGINX trusts an answer; Docker's DNS answers with a TTL of 600 seconds, which would otherwise apply. And proxy_pass takes a variable: NGINX resolves names in a variable at request time through resolver, instead of once at startup. The variable changes one more behaviour. With a variable, proxy_pass passes the request URI through unchanged, and the trailing-slash prefix stripping of the literal form no longer happens, so the rewrite ... break line removes /api itself. A side benefit: NGINX starts even when api does not exist yet, where the literal form fails at startup with host not found in upstream.

Put the fixed configuration back and move the backend again:

Every request succeeded and reached the backend at its new address. The backend's restart took longer than ten seconds here (the health check runs every ten seconds), so NGINX's cached answer had expired by the first request. A move faster than valid can still get 502 or 504 responses for up to ten seconds; lower valid if that matters, at the cost of more DNS lookups. NGINX 1.27.3 and later also offer server api:8000 resolve; inside an upstream block with a zone, which re-resolves in the background and keeps upstream keepalive connections; use it when you need load balancing over several backend addresses.

Renewing the certificate

NGINX reads certificates when it loads its configuration. Renew the files, test the configuration, reload, and check which certificate a client now gets:

Different serial numbers: the reload picked up the new certificate without dropping connections. nginx -t first, always; a reload with a broken configuration is refused, but a restart with one takes the edge down. One trap with single-file bind mounts, which Compose secrets are: OpenSSL rewrote the files in place, so the container saw the new content. A tool that writes a new file and renames it over the old one (many editors, certbot's symlink swap, sed -i) leaves the container looking at the old inode until the container is recreated. Mount a directory instead when your renewal tool works that way, and make it readable by UID 101.

Baking the configuration into an image

For production you may prefer an image that carries its configuration and site, so the image is the whole artifact. Comments in a Dockerfile must start the line; a comment after an instruction is passed to it as arguments:

# Bake the config and the site into the image for production.
# The unprivileged variant runs as UID 101 and listens on 8080/8443.
FROM nginxinc/nginx-unprivileged:1.30-alpine@sha256:15c994d10d6d78658721c3bcafff14cb281fba2a4bdf9d5ba92c416a472516e3
COPY default.conf /etc/nginx/conf.d/default.conf
COPY site/ /usr/share/nginx/html/

The certificate and key still stay out of the image: anyone who can pull an image can read every layer of it. Keep them as secrets or mounts. Clean up:

Try this

Work through “Baking the configuration into an image” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from recipe: nginx as a reverse proxy, keep “Baking the configuration into an image”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related