CoursesDocker in depthLogging & log drivers

Logging & log drivers

Where container output goes, and how to ship it.

Intermediate12 min · lesson 7 of 30

Every office building has a mailroom. You hand over a letter, and someone there decides whether it drops into a pigeonhole downstairs, goes into the building's internal mail system, or gets loaded onto a truck bound for another site. A Docker logging driver does that job for the lines your container prints. When a program inside a container writes to standard output or standard error (stdout and stderr, the two text streams every Unix program prints to), nothing catches those lines on its own. The Docker daemon (dockerd, the background service that actually runs your containers) picks them up and hands each one to the logging driver you configured. From then on, docker logs replays whatever that driver chose to keep. This is why apps in containers should print to stdout and stderr instead of writing to a file buried inside the container. The platform collects the stream it can see, and nothing else. The default driver is json-file, and it behaves perfectly well right up until the day it fills your disk.

terminal
$ docker logs -f --tail 50 web # follow the last 50 lines
127.0.0.1 - - [16/Jul/2026:09:12:03 +0000] "GET /healthz HTTP/1.1" 200 2
127.0.0.1 - - [16/Jul/2026:09:12:04 +0000] "GET /api/orders HTTP/1.1" 200 148
$ docker inspect -f '{{.HostConfig.LogConfig.Type}}' web
json-file

json-file: the default that quietly fills the disk

json-file is named for exactly what it does. Every line your container prints becomes one JSON (JavaScript Object Notation, a plain-text format that stores data as labelled fields) object on its own line, written to a file under /var/lib/docker/containers/<id>/ and tagged with which stream it came from and when. docker logs reads straight out of that file, which is why you can follow output live and see character for character what the app wrote. The problem arrives later. On a stock install, json-file never rotates. No size cap. No cleanup. Ever. One chatty service stuck printing a stack trace in a loop can grow that file to tens of gigabytes over a weekend, and the moment /var/lib/docker is full, the daemon and every container on the node start failing in ways that look nothing like a logging problem. Pulls die. Containers refuse to start. Writes error out. This is the most common Docker logging incident by a wide margin. On a node that is already full, docker inspect --format '{{.LogPath}}' web prints the file doing the damage, and sudo truncate -s 0 on that path hands the space back immediately without stopping the container. Then set the two settings that keep it from happening again.

/etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}

Those two settings are max-size and max-file. Put them in daemon.json once and every container the engine starts from then on inherits bounded logs. max-size caps how large a single log file gets. max-file caps how many log files Docker keeps in total, the live one included, and it deletes the oldest each time a new rotation would push the count over. Ten megabytes across three files means the worst a runaway container can do is about 30 MB instead of the entire disk. Restart the daemon so it picks up the change, then prove it works. Flood a container with output and watch the files roll over.

terminal
$ sudo systemctl restart docker
$ docker run -d --name noisy alpine \
sh -c 'seq 1 2000000 | sed "s/^/flooding the log line /"'
9c1f2a4b7e3d0a5c8b1e6f...
$ ls -lh /var/lib/docker/containers/9c1f2a4b7e3d*/
total 30M
-rw-r----- 1 root root 10M Jul 16 09:20 9c1f...-json.log
-rw-r----- 1 root root 10M Jul 16 09:20 9c1f...-json.log.1
-rw-r----- 1 root root 10M Jul 16 09:20 9c1f...-json.log.2
$ docker logs --tail 1 noisy
flooding the log line 2000000

local: same place, better manners

Docker ships a second on-node driver called local, and it is the one you would reach for if json-file were not already wired in as the default. It stores logs in a compact binary format rather than JSON text, so the same output takes noticeably less room on disk. It also rotates by itself. Out of the box, local holds up to 100 megabytes spread over five 20-megabyte files and gzips (compresses) the rotated ones, so you get bounded, compressed logs without touching a single option. docker logs still works, because local keeps everything on the node the same way json-file does. Building a node from scratch? Setting local as the default is a cleaner starting point than capping json-file after the fact.

terminal
$ docker run -d --name svc --log-driver local \
alpine sh -c 'seq 1 2000000 | sed "s/^/hello from local /"'
b7e3d0a5c8b1e6f2a9d4c7...
$ docker inspect -f '{{.HostConfig.LogConfig.Type}}' svc
local
$ ls -lh /var/lib/docker/containers/b7e3d0a5c8b1*/local-logs/
-rw-r----- 1 root root 20M Jul 16 09:34 container.log
-rw-r----- 1 root root 1.2M Jul 16 09:34 container.log.1.gz
-rw-r----- 1 root root 1.2M Jul 16 09:34 container.log.2.gz
$ docker logs --tail 1 svc
hello from local 2000000

journald: let the host's journal hold it

On a host running systemd (the service manager that starts and supervises programs on most Linux distributions), you can skip Docker's own files and send container output to the system journal, the same place logs from sshd (the SSH login service) and cron (the scheduled job runner) already live. The driver is called journald. Once it is set, you read logs with journalctl and filter by container name or ID (identifier), so container output and host output sit in one queryable stream with real timestamps and structured fields. The journal enforces its own size limit and vacuums old entries, so it will not run away the way an uncapped json-file does. docker logs keeps working here too, because the journal lives on the same box.

terminal
$ docker run -d --name api --log-driver journald alpine \
sh -c 'while true; do echo "api heartbeat $(date +%T)"; sleep 5; done'
d0a5c8b1e6f2a9d4c7b3e0...
$ journalctl CONTAINER_NAME=api -o cat --no-pager | tail -3
api heartbeat 09:41:07
api heartbeat 09:41:12
api heartbeat 09:41:17
$ docker logs --tail 1 api
api heartbeat 09:41:17
Which driver for this node?
Where should the logs end up?
pick per node, set it in daemon.json
read on the box
json-file (capped) or local
docker logs works; local rotates and gzips by default, json-file needs max-size/max-file
unify with host logs
journald
query with journalctl alongside sshd and cron; the journal handles its own rotation
must survive the node
fluentd / syslog / gelf / awslogs
ships off-box to a collector or a SIEM (security information and event management) system; docker logs holds only a short local cache, so the real history lives there
Logs kept on the node die with the node, and they are the first thing an intruder edits. Anything you need for an audit or an incident belongs off the host.
A dead collector breaks two different ways
Which way depends on whether the driver opens a connection at all. fluentd, and syslog or gelf pointed at a tcp:// address, dial the collector when the container starts, so with nothing listening the container can hang or refuse to come up. At least that tells you something is wrong. Point gelf at the udp:// address it is usually configured with (UDP, User Datagram Protocol, throws packets at an address without ever checking that anyone is on the other end) and you get the quieter failure: there is no connection to fail, so the container starts perfectly and drops every line into a void with no error anywhere. Set fluentd-async to true (or whatever the connecting driver calls its equivalent) so a logging hiccup cannot take the app down with it, and give it a buffer so a brief collector outage does not lose lines. For the UDP drivers the only warning you will ever get is one you build yourself, so alert on the collector going quiet rather than trusting the container to complain. json-file and local fail neither way, which is why plenty of teams keep a capped on-node driver and forward from there instead of logging straight onto the network.

Stdout is the contract

Docker hands the container a pipe rather than a terminal, and a runtime that buffers its output whenever it is not talking to a terminal will look completely silent under docker logs until a few kilobytes have piled up. Python is the usual offender, PYTHONUNBUFFERED=1 the usual fix. Write only to a file inside the container's writable layer and the lines are invisible to docker logs and to most collectors, unless you bolt on a sidecar container or a tailer running on the host to go and read them. json-file is what you get by default, and it will fill a disk unless max-size and max-file are set. journald, syslog, fluentd and the cloud drivers each trade a little local convenience for logs that live somewhere central.

During an incident, the first question to ask is where the driver put the bytes. docker logs reads the driver's store and nothing else, so if the app wrote to /var/log/app.log inside the container, those lines will not be there no matter how long you stare at the screen. Print structured single-line JSON to stdout when a parser has to read it, keep secrets out of log lines (they land on disk, in the journal and in the collector), and do not count on the driver to tidy up multiline stack traces. json-file, local and journald have no option for that, so a forty-line Java trace arrives as forty separate lines and it is your collector's parser that stitches them back together; awslogs is the rare exception, with awslogs-multiline-pattern. journald will also drop lines that arrive faster than its own rate limit, and you raise that in journald.conf on the host, not anywhere in Docker.

Choosing between an on-node driver and journald is a trade between what your operators already know and what stays identical across every node. Pick one standard for the platform, set it in daemon.json, and override it per container only for the genuinely noisy exceptions. When a disk alert fires on /var/lib/docker, unbounded container logs are a likelier cause than a pile of images.

Write the change down before you touch the node. Which driver was in force, which one you are moving to, the exact log-opts, and the one edit that puts it back. The docker inspect check shown above prints the driver for a running container, so paste the before and the after into the ticket. A driver change only reaches containers created after the daemon restart, so record which workloads were recreated and which are still running on the old setting. That list is what tells the next person whether a container with a swelling log file was missed in the rollout or left uncapped on purpose.

Try this

Run this on a lab engine (Docker 24 or newer is fine). The caps go on the container here rather than the daemon, so nothing else on the host changes. The container floods its own log on purpose, so you watch a rollover happen instead of taking the caps on trust.

terminal
$ docker run -d --name logdemo --log-opt max-size=1m --log-opt max-file=2 \
alpine:3.20 sh -c 'seq 1 200000 | sed "s/^/logdemo line /"'
4f2c9d1a8b3e07c5d2f6a...
$ docker inspect logdemo --format '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}'
json-file {"max-file":"2","max-size":"1m"}
$ docker inspect logdemo --format '{{.LogPath}}'
/var/lib/docker/containers/4f2c9d1a8b3e/4f2c9d1a8b3e-json.log
$ sudo ls -lh /var/lib/docker/containers/4f2c9d1a8b3e/ | grep json.log # max-file=2 keeps two files, not three
-rw-r----- 1 root root 1.0M Jul 16 09:52 4f2c9d1a8b3e-json.log
-rw-r----- 1 root root 1.0M Jul 16 09:52 4f2c9d1a8b3e-json.log.1
$ docker logs --tail 1 logdemo
logdemo line 200000

Takeaway

Print to stdout and stderr, cap json-file with max-size and max-file or move the node to local, and treat docker logs as a view of the driver's copy rather than the whole truth. The command worth keeping in your head is docker inspect -f '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}}' on any node you have not touched before, because it answers in one line whether that box is still on the uncapped default.

Quick check
01You add "max-size": "10m" and "max-file": "3" to the log-opts in /etc/docker/daemon.json, run sudo systemctl restart docker, and a week later the log file for the web container is still well past 40 MB. docker inspect -f '{{.HostConfig.LogConfig.Type}}' web prints json-file. What happened?
Correct — The driver and its options are read when a container is created, so web is still on the settings it started with. Recreate it and the cap takes effect.
Incorrect — max-size caps one file and max-file caps how many are kept, so 10m with 3 files means about 30 MB in total rather than one 30 MB file.
Incorrect — daemon.json is exactly the right place to set this once so every new container inherits it. Per-container --log-opt is for the genuinely noisy exceptions.
Incorrect — json-file rolls over while the container is still running. The flood test in the lesson shows three 10M files under /var/lib/docker/containers/ with the container mid-flight.
02You move two services off json-file. One gets --log-driver fluentd, the other gets gelf pointed at a udp:// address. The collector behind both is down. What do you expect to see?
Incorrect — fluentd does open a connection at startup, so that half holds. gelf over UDP never opens one, so there is no connection for it to fail on.
Correct — The drivers that dial out tell you loudly that something is broken. The UDP one is the dangerous half, because it looks healthy while the lines go nowhere.
Incorrect — A buffer is something you ask for on a connecting driver, not a default you inherit. Neither of these was given one, and the UDP side has nothing to hold onto.
Incorrect — UDP throws packets at an address without ever checking that anyone is listening, so no error comes back. That silence is what makes it the quieter failure.
03A service runs with --log-driver local. A teammate lists /var/lib/docker/containers/<id>/local-logs/, sees container.log next to container.log.1.gz, and says the logs are unreadable now so the node should go back to json-file. What do you tell them?
Incorrect — journalctl is how you read the journald driver, filtering with something like CONTAINER_NAME=api. local keeps its own files on the node and docker logs reads them directly.
Incorrect — Compressing rotated files is local's habit, not json-file's. max-file only decides how many log files json-file keeps before it deletes the oldest one.
Correct — docker logs --tail 1 svc prints hello from local 2000000 just as you would expect. The binary format describes how the bytes sit on disk, not what comes back to you.
Incorrect — A stock json-file driver does not rotate at all until you cap it. The 100 MB spread over five 20 MB files is what local gives you with nothing configured.

Related