Recipe: Redis

Cache or datastore: persistence, memory limits and ACL users.

Start the official Redis image with no configuration and ask it a question from another container:

protected-mode is no, and a client that knows nothing but the container's name wrote a key. Redis itself has protected mode on by default: with no password and no explicit bind, it refuses clients that do not come from loopback. The official image switches that off, because in a container every client arrives over a network interface and the default would make the image unusable. Redis notices and logs the warning above, which few people read. So docker run -p 6379:6379 redis on a host with a public address is an open database on the internet, reachable by anything that scans for port 6379. Whatever this recipe adds, it has to replace protected mode, not rely on it.

Everything runs on the main lab VM, secopslog-docker, as ubuntu in ~/lab/r-redis. The lesson files download has redis.conf, compose.yaml, make-acl.sh and rcli.

Cache or datastore

Decide this first, because it sets most of the configuration:

Eviction is silent: the client that triggers it gets OK, and the keys that made room simply stop existing. That is right for a cache and wrong for anything that exists only in Redis. This recipe builds the datastore variant; the memory section at the end shows what changes when the same instance is switched to cache behaviour.

Access control with an ACL file

requirepass sets one password for the default user, and every client then has full power, including FLUSHALL and CONFIG SET. ACL users (Redis 6 and later) are better: each client gets a name, a password and only the commands and keys it needs. This recipe defines three users and writes them to an ACL file with a small script:

#!/bin/sh
# Writes secrets/users.acl from the two password files. The ACL file stores SHA-256 hashes only.
set -eu
hash() { printf %s "$(cat "$1")" | sha256sum | cut -d' ' -f1; }
{
  echo "user default off"
  echo "user app on #$(hash secrets/app_password.txt) ~app:* -@all +@read +@write +@keyspace +ping -@dangerous"
  echo "user ops on #$(hash secrets/ops_password.txt) ~* &* +@all"
} > secrets/users.acl

default is switched off, so a client that does not log in can do nothing. app may read, write and expire keys whose names start with app: and nothing in the @dangerous category (FLUSHALL, CONFIG, KEYS, DEBUG and the like), plus PING for the health check. ops is the administrator. A # followed by a SHA-256 hash stores the password's hash instead of the password, so the ACL file can be read without revealing either password. Create the passwords and the ACL file:

The hashes differ on every run. The files are 0644 in a 0700 directory: Redis runs as the redis user (UID 999) and reads the ACL file itself, and a Compose secret without Swarm is a bind mount that keeps the host file's owner and mode, so 0600 would lock Redis out ("Recipe: PostgreSQL, MySQL and MongoDB" shows that failure for Postgres and Mongo).

The configuration

# Redis as a small datastore: writes are journaled and nothing is evicted silently.
# appendonly is set on the command line in compose.yaml (an RDB restore needs it off for one start).
aclfile /run/secrets/redis_users_acl
appendfsync everysec
maxmemory 64mb
maxmemory-policy noeviction

name: lab-cache
 
services:
  redis:
    image: redis:8-alpine
    # set REDIS_APPENDONLY=no only for the first start of an RDB restore
    command: ["redis-server", "/usr/local/etc/redis/redis.conf", "--appendonly", "${REDIS_APPENDONLY:-yes}"]
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
      - redisdata:/data
    secrets:
      - redis_users_acl
      - redis_app_password
      - redis_ops_password
    mem_limit: 256m
    healthcheck:
      test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(cat /run/secrets/redis_app_password)\" redis-cli --user app ping | grep -q PONG"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 30s
      start_interval: 1s
    # time to finish the AOF/RDB write on shutdown before Docker sends SIGKILL (default 10s)
    stop_grace_period: 1m
    restart: unless-stopped
    # no ports: services on this project's network reach it as redis:6379
 
volumes:
  redisdata:
 
secrets:
  redis_users_acl:
    file: ./secrets/users.acl
  redis_app_password:
    file: ./secrets/app_password.txt
  redis_ops_password:
    file: ./secrets/ops_password.txt

aclfile points at the secret. appendfsync everysec is the default, written down because it is the durability promise: the append-only file (AOF) is flushed to disk once a second, so a power loss costs at most about a second of acknowledged writes. always flushes on every write and is much slower; no leaves it to the kernel. maxmemory 64mb with noeviction is the datastore choice from the table. appendonly is passed on the command line instead, so that one restore step below can switch it off without editing the file.

The command starts with redis-server. The image's entrypoint drops from root to the redis user (with setpriv, keeping no capabilities) when it starts the server, and it also recognises a first argument ending in .conf, so a bare config path works too; a command that starts with sh -c would skip the drop and run the server as root. The entrypoint also fixes ownership of files in /data and of the config file when the redis user cannot read them, which fails harmlessly on this read-only mount and is one more reason to make the file readable. The health check logs in as app and greps for PONG, because redis-cli exits 0 even when the server answers with an error such as NOAUTH. mem_limit: 256m leaves the server room above maxmemory, for reasons the memory section shows. And there is no ports: entry: services in this project reach Redis as redis:6379 on the project network, and nothing else can.

Typing a password for every redis-cli call is clumsy and passing one on a command line leaks it into process listings, so a small wrapper runs redis-cli inside the container as an ACL user, with the password read from the secret file there:

#!/bin/sh
# usage: ./rcli USER ARGS...  runs redis-cli in the redis container as the ACL user USER.
# The password is read from the secret file inside the container, so it never appears in a host
# command line or in redis-cli's arguments.
user=$1; shift
exec docker compose exec -T redis sh -c \
  'REDISCLI_AUTH="$(cat "/run/secrets/redis_${0}_password")" exec redis-cli --user "$0" "$@"' "$user" "$@"

Up, and who can do what

The PORTS column shows 6379/tcp with no host address or arrow: exposed inside the network, published nowhere. A client container on the project network reaches the server by name and gets NOAUTH, and exit=0 shows redis-cli reporting success for it, as promised. The app user writes and reads its own keys and is refused a key outside app:, CONFIG and FLUSHALL. Through ops, the server confirms the settings: 64 MiB, noeviction, AOF on. protected-mode is still no, which no longer matters because nothing is reachable without a password. The server process runs as redis.

If a tool on the host itself needs Redis, publish the port on loopback only, 127.0.0.1:6379:6379, and keep the ACL users. A bare 6379:6379 publishes on every address of the host, and the Docker-managed firewall rules for published ports take effect before most host firewall setups ("Publishing ports and the packet path" explains why ufw does not help there).

Persistence: AOF and RDB

The key survived a restart because Redis replayed the AOF on start. Since Redis 7 the AOF is a directory: a base file in RDB format, an incremental file with the writes since, and a manifest that ties them together. dump.rdb is a point-in-time snapshot, written on the default schedule (after 1 hour if 1 key changed, 5 minutes if 100 changed, 1 minute if 10,000 changed) and on shutdown. That final write on shutdown is why the Compose file sets stop_grace_period: 1m: with a large dataset it can take longer than Docker's default ten seconds, after which Docker sends SIGKILL and the write is lost. Restarts load the AOF when it is enabled and ignore dump.rdb. That rule matters in a moment.

Backup and restore

redis-cli --rdb asks the server for a fresh snapshot over the replication protocol and writes it out, here to stdout and into a host file. It needs a user allowed to run SYNC, which is why it runs as ops:

The progress messages go to stderr; the file starts with the REDIS magic and the RDB format version. Sizes vary. Now the trap. Lose the volume, copy the backup into a fresh one as dump.rdb, and start the normal configuration:

The key is gone (the empty line is the reply for a missing key). With appendonly yes and no AOF in the volume, Redis created a new, empty AOF base on start and loaded that instead of dump.rdb. Nothing failed and the health check was green. The working procedure starts the server once with AOF off, so it loads the snapshot, then turns AOF on at run time, which writes a new AOF from memory:

The first GET shows the snapshot loaded. CONFIG SET appendonly yes started an AOF rewrite, the loop waited until it finished, and docker compose up -d recreated the container with the normal command line, which loaded the new AOF. Keep the backup until that last check passes. Copying a stopped instance's whole /data (AOF directory included) is the other valid backup, with the stop-copy-start method from "Volumes, bind mounts and tmpfs in practice".

Memory: maxmemory, the policy and the container limit

Fill the instance past its 64 MiB with 400,000 keys of 200 bytes each, using redis-cli --pipe, which streams commands without waiting for each reply:

With noeviction, Redis accepted writes until it reached maxmemory and then answered every further write with an OOM error, which --pipe counts in its summary (the exact split varies between runs). The application's next SET failed loudly too. That is the datastore behaviour you want: a failed write reaches someone, a silently evicted job does not. Switch the same instance to cache behaviour:

Now the write succeeds and evicted_keys counts the keys Redis dropped to make room. Redis samples a few keys per eviction (maxmemory-samples, 5 by default) rather than keeping an exact LRU order, so which keys go is approximate. Watch evicted_keys on a cache: a steady climb means the working set is bigger than maxmemory.

The container limit is separate from maxmemory and must be larger. Redis's memory use is more than its data: allocator fragmentation, client buffers, and above all the copy-on-write cost of a fork, because RDB snapshots and AOF rewrites run in a child process that shares memory with the parent until the parent changes a page. Under heavy writes during a rewrite, the total can approach twice the dataset. Set maxmemory to roughly half to two thirds of the container limit. If the kernel's memory limit is hit first, Redis is killed instead of answering with errors, and everything since the last fsync is lost. Redis also logs a warning when the host has vm.overcommit_memory at 0, because a fork can then fail under memory pressure; that is a host sysctl, set on the Docker host (not in the container), and the lab VM leaves it unchanged.

Clean up:

Try this

Work through “Memory: maxmemory, the policy and the container limit” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from recipe: redis, keep “Memory: maxmemory, the policy and the container limit”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related