Scheduling: cron and systemd timers

crontab, cron environment and timers.

Servers do much of their work on a schedule: rotating logs, refreshing package lists, taking backups, checking disks. Linux has two schedulers for this, the classic cron daemon and systemd timers, and you will meet both on every Ubuntu and RHEL server. This lesson shows how to write a crontab and check that it really runs, why a job that works at your prompt fails under cron, how to write a systemd timer and what Persistent= really does, and how to choose between the two. Scheduled jobs are also a favourite hiding place for attackers, so knowing where schedules live is part of knowing a server.

Writing and installing a crontab

A crontab is a table of jobs for the cron daemon. Each line has five time fields and then a command: minute (0 to 59), hour (0 to 23), day of the month (1 to 31), month (1 to 12, or jan to dec) and day of the week (0 to 7, where both 0 and 7 are Sunday, or mon to sun). A field can be * (every value), a list such as 1,15, a range such as 1-5, or a step such as */15 (every fifteenth value). So 30 2 * * * is 02:30 every day, */15 * * * * every fifteen minutes, and 0 8 * * mon-fri 08:00 on weekdays. Shortcuts replace all five fields: @hourly, @daily, @weekly, @monthly, and @reboot, which runs when the cron daemon starts.

Two rules surprise people. When both day fields are restricted, a job runs when either matches: 0 9 1-7 * mon runs on each of the first seven days and on every Monday, rather than on the first Monday alone. And % in a command is special: cron turns the first % into the end of the command and sends the rest as input, so it must be written \%. A comment must be on a line of its own; anything after the command, including #, is passed to the shell.

The examples need a directory for crontab files and a small script, disk-report, that prints how full the root filesystem is. printf writes the script's two lines (\n ends a line) and chmod +x makes it executable:

Ubuntu's ~/.profile adds ~/.local/bin to PATH at login only if the directory already exists, so if you have just created it, log out and in again. Here is a crontab for the deploy account with three jobs that run every minute, a convenient schedule for testing. The first saves the environment the job runs in, the second appends the time to a file, and the third runs disk-report:

# m h dom mon dow  command
* * * * *  env > $HOME/cron/env.txt
* * * * *  echo "ran at $(date +%H:%M)" >> $HOME/cron/percent.log
* * * * *  disk-report >> $HOME/cron/report.log 2>&1

Each user has their own crontab, installed with the crontab command. crontab -l prints yours, crontab -e opens it in your editor and installs it when you leave the editor, and crontab FILE replaces it with a file, which is easy to keep under version control. crontab -r removes it without asking, and -i makes it ask first. Both crontab FILE and crontab -r act on the whole table, so on an account that already has jobs, save them first with crontab -l > ~/crontab.bak. This account has none yet:

crontab checks a file before installing it and refuses some mistakes, such as the misspelt shortcut in this one:

# m h dom mon dow  command
@dayly  $HOME/.local/bin/disk-report

It does not catch everything. This file schedules a job for minute 60 of hour 25, which never exists:

# m h dom mon dow  command
60 25 * * *  $HOME/.local/bin/disk-report

crontab installed it without complaint, and when the next minute started, the cron daemon logged nothing about it either. Minute 60 and hour 25 never come, so the job is never due, and nothing anywhere tells you. A syntax check is not a test: read a new schedule back with crontab -l, and check the log after the first time the job should have run. The files themselves live in /var/spool/cron/crontabs on Ubuntu, one per user, readable only by their owner. Do not edit them there: the crontab command installs them, and cron notices the change within a minute.

Why a job works at the prompt and fails under cron

At the prompt, all three commands work. disk-report is found because Ubuntu's ~/.profile adds ~/.local/bin to PATH when you log in:

A minute after the crontab was installed, the results are different:

percent.log was never created, and report.log holds an error from /bin/sh. env.txt shows why disk-report was not found: cron runs jobs with /bin/sh, not bash, in the home directory, with a short environment. It does not read ~/.profile or ~/.bashrc. PATH comes from /etc/environment, which Ubuntu's PAM configuration for cron loads, so system commands in /usr/bin and /usr/sbin are found, but ~/.local/bin, tool managers that add themselves in ~/.bashrc, and any variable you export there are missing. The job processes log under the name CRON, so journalctl -t CRON shows what happened to the other job:

cron logs each job it starts with CMD, and the percent.log job was cut off at the %: the shell received an unfinished command and failed. Its error message is the job's output, and cron mails a job's output to the crontab's owner, or to the address in a MAILTO= line (MAILTO="" turns mail off). Ubuntu Server installs no mail server by default, so cron logs No MTA installed, discarding output and the error is gone. That is the most common reason a cron job "does nothing": it fails, and nobody sees why. (Select these lines with -t CRON rather than -u cron: a job process that exits within milliseconds can be gone before journald records which unit it belonged to, and -u then misses its lines.)

The fix is to use full paths for your own tools, escape %, and send the output somewhere you will read it. logger -t NAME writes it to the journal and syslog under a name you can search for:

# m h dom mon dow  command
* * * * *  echo "ran at $(date +\%H:\%M)" >> $HOME/cron/percent.log
* * * * *  $HOME/.local/bin/disk-report 2>&1 | logger -t disk-report

System crontabs

Besides the per-user crontabs, cron reads /etc/crontab and every file in /etc/cron.d. These are edited directly by administrators and packages, and their lines have one more field, the user to run the job as, between the schedule and the command:

PATH is commented out because this cron (Debian's, as used by Ubuntu) takes it from its environment. The four jobs run every script in /etc/cron.hourly, cron.daily, cron.weekly and cron.monthly with run-parts, the daily ones at 06:25; the test -x /usr/sbin/anacron part hands them to anacron instead when it is installed, which it is not on Ubuntu Server by default. RHEL uses a different cron, cronie, whose service is crond and whose log is /var/log/cron. Its hourly scripts are started from /etc/cron.d/0hourly, and the daily, weekly and monthly ones by anacron, which runs any of them that is overdue, for example because the machine was off:

systemd timers

A systemd timer is a unit that starts another unit on a schedule. The job is an ordinary service, usually Type=oneshot (it runs to completion and exits), and the timer, with the same name and the suffix .timer, says when. Both files go in /etc/systemd/system, as in "Services with systemd":

[Unit]
Description=Disk usage report
 
[Service]
Type=oneshot
ExecStart=/usr/bin/df -h --output=target,pcent /

[Unit]
Description=Disk usage report every day at 02:30
 
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
 
[Install]
WantedBy=timers.target

OnCalendar uses systemd's calendar format, weekday (optional), year-month-day hour:minute:second, with * for any value. systemd-analyze calendar checks an expression and shows when it next elapses, which is the easiest way to get one right. Unlike crontab, it rejects values that cannot exist:

You enable and start the timer, not the service. systemctl list-timers then shows when it fires next (NEXT, LEFT) and when it last ran (LAST, PASSED):

Without a unit name, systemctl list-timers lists every active timer. A fresh Ubuntu Server has sixteen, which refresh package lists, rotate logs, trim and check filesystems, rebuild the man database and collect sysstat figures:

Each run is logged under the service's name, so journalctl -u ess-cron-report.service shows its output and exit status. sudo systemctl start ess-cron-report.service runs the job once, straight away, which is how you test it.

What Persistent= really does

A calendar timer fires only while its timer unit is active, and a timer is inactive while the machine is switched off. Persistent=true makes systemd record the time of each run in a file under /var/lib/systemd/timers; when the timer starts again, at boot for example, systemd runs the service at once if a scheduled time passed while the timer was inactive. The default is false, and then a missed run is simply skipped until the next scheduled time. It applies only to OnCalendar timers.

Each file is empty; its modification time is the record of the last run. The lab cannot switch the machine off in the middle of a lesson, so it did the equivalent: it stopped the timer and set the date on its stamp file two days back, which is what the file looks like when the server has been off over the scheduled time. Then the timer is started, as it would be at boot:

The report ran immediately instead of waiting for 02:30 the next day, and LAST now shows that run. Suspend is different from power-off: a calendar timer that elapses while the machine is suspended runs when it resumes, with or without Persistent=. Packages use these settings too:

OnCalendar=daily means midnight. RandomizedDelaySec=1h adds a random delay of up to an hour, chosen again for each run, so that a fleet of servers does not rotate logs or download updates at the same second; that is why logrotate.timer in the default list above fires at 00:13:23 rather than at midnight. Timers are also accurate only to a minute by default (AccuracySec=), so systemd can group wake-ups.

Choosing, and knowing what is scheduled

Use a timer for system jobs: the output is in the journal without any redirection, Persistent= makes up missed runs, the job can depend on other units, and every setting from "Services with systemd" (a dedicated user, resource limits, sandboxing) applies to it. A timer also never runs two copies of a job at once: if the service is still running when the timer elapses, systemd leaves it running and starts nothing new. cron starts a new copy every time the schedule matches, so a */5 job that sometimes takes seven minutes piles up, with two copies writing the same files. For such a job in a crontab, flock -n ~/cron/sync.lock COMMAND runs the command only if no other copy holds the lock file, and otherwise exits at once. cron is still fine for a simple job in your own account, needs no root, and is the same on every Unix. Whichever you use, test the schedule with a short interval or systemctl start, and check the log after the first real run.

Try this

Save any crontab you already have with crontab -l > ~/crontab.bak, because the commands below replace and then remove the whole table. Install a crontab with one job that runs every two minutes and logs to the journal: echo '*/2 * * * * df -h / | logger -t diskcheck' | crontab - (- reads the crontab from the pipe). Check it with crontab -l, wait until the next even minute, and read journalctl -t diskcheck: you should see the two lines of df output. df and logger need no full path, because both are in /usr/bin, which is in cron's PATH. Then find the timer expression for the same schedule with systemd-analyze calendar --iterations=3 '*:0/2', which lists the next three even minutes. Finish with crontab -r, or with crontab ~/crontab.bak to put your old jobs back.

Takeaway

Write jobs so that they cannot fail silently: full paths for your own tools, \% instead of %, and output sent to the journal or a log. After every change, check journalctl -t CRON or systemctl list-timers once the job should have run, and prefer a timer with Persistent=true for anything the server must not miss or must never run twice at once.

Related