Resource control with slices & cgroups

Cap CPU, memory, and IO per service.

A single busy service should never be able to take the whole machine down with it. One process that eats every byte of memory, or pins all the cores at 100%, or floods the disk with writes, can starve everything else on the box: your logging agent, your monitoring, the SSH daemon (the program that lets you log in from another machine), the very tools you would reach for to fix the mess. On a Linux host running systemd (the first process the kernel starts, process ID 1, which then launches and supervises every other service), you can put a firm ceiling on what each service is allowed to take. You write the ceiling into the service's unit file, and the kernel holds the line. No containers, no sidecar, no outside agent.

The machine is like an apartment building, and each running service is a tenant. Left unchecked, one tenant could run every tap, every light, and every radiator at full blast and leave nobody else any water or power. So the building meters each unit: this apartment gets at most so much water pressure, so much electricity, so many amps before its breaker trips. That kind of metering already exists on your Linux box. It is called the cgroup (control group), and systemd has quietly wrapped one around every service you run.

Every service already runs in a cgroup

A cgroup is a labelled box the kernel (the core part of the operating system that talks directly to the hardware) draws around a set of processes so it can count and cap what they use together: processor time, memory, and disk input and output (reading from and writing to storage). The meter is already installed. You do not have to build anything to start measuring. Ask systemd about a running service and it shows you the box and what is inside it.

Read the last four lines like a dashboard. Tasks: 12 (limit: 100) is the number of processes and threads in the box against the cap you set. Memory: 480.2M (high: 400.0M max: 512.0M) is live usage against two different memory limits (the difference between them matters, and we get to it below). CGroup: /system.slice/myapp.service is the box's address in the tree. On any current Debian or Ubuntu release that tree is the cgroup version 2 unified hierarchy, one tidy structure the kernel exposes as plain files under /sys/fs/cgroup.

Setting the caps in the unit

You set the limits declaratively, as ordinary directives in the [Service] section of the unit. The service does not have to know or cooperate. The kernel enforces the numbers whether the code likes it or not.

[Service]
ExecStart=/usr/local/bin/myapp
# at most half of one core's worth of CPU time
CPUQuota=50%
# soft cap: throttle and reclaim above this, no kill
MemoryHigh=400M
# hard cap: a process is OOM-killed if the service crosses this
MemoryMax=512M
# do not let it escape the cap into swap
MemorySwapMax=0
# ceiling on processes + threads (blunts fork bombs)
TasksMax=100
# relative disk-IO priority (default is 100)
IOWeight=50

A quick warning about the comment lines above: in a systemd unit each # has to sit on its own line. systemd does not understand a comment tacked onto the end of a directive, so IOWeight=50 # priority would fold that whole trailing text into the value and break the setting. Keep comments on their own lines, as here. Now the settings themselves. CPUQuota=50% means the service gets at most half of one core's time, even on a machine with 64 of them. MemoryHigh and MemoryMax are the two memory limits you saw in the status output; the hard one is enforced by the kernel's OOM (out-of-memory) killer, the code that frees memory by terminating a process when a limit is hit. TasksMax caps how many processes and threads the service can spawn, which blunts a fork bomb (a process that copies itself again and again until the system runs out of slots to track anything). IOWeight is a relative share of disk bandwidth when several services want the disk at once: 50 against the default of 100 means this service yields to normal services under contention. It is a weight, not a hard ceiling; for an absolute cap you set IOReadBandwidthMax and IOWriteBandwidthMax against a specific device.

systemctl show gives you the effective values the manager is actually enforcing, which is what you want in an audit: not what the file claims, but what is loaded right now. CPUQuota=50% shows up as CPUQuotaPerSecUSec=500ms, meaning 500 milliseconds of CPU time for every second of real time. The memory numbers are in bytes (512M is 536870912). To see it straight from the kernel, read the control files under the cgroup directly.

cpu.max reads 50000 100000, which is a budget and a window: 50000 microseconds of CPU time out of every 100000 microsecond window, exactly half a core again. memory.current is what the service is using this second (503455744 bytes, about 480 mebibytes), sitting between the soft cap and the hard cap where you want it.

Slices: a shared budget for a whole category

Capping one service is fine until you have thirty of them. Say you run a pile of overnight batch jobs (imports, report builds, backups) and you want the whole category held under, for example, two cores and two gigabytes no matter how many happen to run at once. That is a slice. A slice is a department budget: instead of handing each person a separate limit, you give the department one pool and let its members share it.

systemd already sorts everything into a handful of top-level slices. System services live under system.slice. Interactive logins live under user.slice, one sub-slice per human. Virtual machines and containers live under machine.slice. All of it hangs off the root, which systemd calls -.slice. Because cgroups are a tree, a limit you set on a slice caps everything beneath it, added together. You can make your own slice and drop services into it.

[Unit]
Description=Overnight batch jobs, collectively capped
 
[Slice]
# two cores total, shared by everything in here
CPUQuota=200%
# 2 GiB hard cap for the whole group
MemoryMax=2G
# low disk priority so batch never starves live traffic
IOWeight=20

Then point services at it with one line, Slice=batch.slice, in each unit's [Service] section. Now import.service and report.service and any others share the pool. Ten of them running together still cannot exceed two cores or two gigabytes between them. Look at the tree with systemd-cgls.

Watching it live, and changing a limit without a reboot

systemd-cgtop is top for cgroups: the same rolling table of who is using what, but grouped by service and slice instead of by raw process. When someone pages you with "the box is on fire," this is the one command that answers which service, by name, in seconds.

The shape of an incident jumps out here. A single service pinned near its CPU cap is doing its job under load. A slice climbing toward its memory ceiling means the whole category is about to get squeezed. And a service you do not recognise chewing CPU inside system.slice is exactly what a defender wants to catch early: a coin miner or a rogue task shows up as an unfamiliar cgroup burning resources it has no business burning.

You can also change a limit on a running service immediately, with no edit-and-restart cycle.

That takes effect at once. By default set-property also writes a small drop-in file under /etc/systemd/system.control/ so the change survives a reboot. If you want the change only for right now, for firefighting, add --runtime and it evaporates on the next boot. Permanent policy belongs in the unit file in version control; set-property is for the 3 a.m. intervention.

The record the limits leave behind

Every throttle and every kill is counted, and the tally is a small forensic ledger you can read after the fact. The kernel keeps a running count per cgroup in memory.events.

high 8423 means the service was pushed against its soft cap thousands of times: constant back-pressure, a strong hint it genuinely wants more memory than you gave it. oom_kill 4 means the hard wall actually killed a process four times. Cross-check against the journal to watch systemd record each kill in its own words.

For an operator that is the difference between "the app is flaky" and "the app is being killed by a limit you set." For a defender, memory.events and the cgroup tree together make a cheap tripwire: a service that suddenly starts hitting caps it never used to, or a brand-new cgroup you did not deploy, is worth a hard look.

Capping something before you trust it

The same machinery works on a process that is not a service yet, which helps when you have a binary you do not fully trust and want to watch it without handing it the run of the host. systemd-run wraps a one-off command in its own transient cgroup with whatever limits you name.

Now the unknown binary can burn at most a quarter of a core and 200 mebibytes, cannot spill into swap to dodge the memory cap, and cannot fork past 64 tasks. If it tries to run away, it hits the wall you built instead of the machine's, and when it exits the transient cgroup disappears. This does not fence off what the process can touch on disk or the network, so it is no substitute for real isolation, but as a blast-radius limit on something you are about to run, it costs one line.

Make MemoryHigh-below-MemoryMax plus a TasksMax your default for anything that faces untrusted input, and read memory.events on that service the next morning. If high is climbing into the thousands, the service is asking for more room than you gave it; if oom_kill is above zero, your backstop is doing real work, and you get to decide whether that is protection you designed or an outage waiting to happen.

Try this

Work through “Capping something before you trust it” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: memoryMax kills; MemoryHigh throttles. Know which you set. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related