Run a script on a schedule with cron, and stop two runs overlapping

cron is the oldest and simplest job scheduler on Unix, and it is still how a great deal of real work gets run. Its simplicity is also why a first attempt often fails silently. This guide covers the entry itself, the reasons jobs "do not run", and how to stop a slow job from being run twice at once.

1. The format

Edit your own crontab with:

crontab -e

Each line has five time fields and then a command:

# minute  hour  day-of-month  month  day-of-week  command
  */15    *     *             *      *            /usr/bin/python3 /opt/jobs/report.py
  30      6     *             *      1-5          /opt/jobs/morning.sh
  0       2     1             *      *            /opt/jobs/monthly.sh

• */15 * * * *: every 15 minutes.
• 30 6 * * 1-5: 06:30 on Monday to Friday (0 or 7 is Sunday).
• 0 2 1 * *: 02:00 on the first day of every month.

crontab -l lists what is installed. The time zone is the system's, unless the cron implementation supports setting CRON_TZ, so check which you are in.

2. Why a job works by hand and not in cron

Cron runs jobs in a bare environment, not your login shell. The usual causes:

• A short PATH. Use absolute paths for the program and for the script (/usr/bin/python3, not python3), and for any files the script opens, or set the working directory first: cd /opt/jobs && ....
• No profile or virtualenv. Call the virtual environment's Python directly: /opt/jobs/.venv/bin/python.
• Missing environment variables. Define them at the top of the crontab, or in a file your script reads, rather than assuming your shell's.
• The % character. In a crontab, an unescaped % is treated as a newline. Escape it as \%, which matters in date +%F style commands.
• Permissions. The script must be executable (chmod +x) and readable by the user whose crontab it is.

3. Capture the output

By default cron tries to e-mail a job's output to the user, which on most machines goes nowhere. Redirect it to a log yourself:

*/15 * * * * /opt/jobs/.venv/bin/python /opt/jobs/report.py >> /var/log/report.log 2>&1

>> appends, and 2>&1 sends error messages to the same place. Without it, a failing job leaves no trace. Rotate the log file (for example with logrotate) so it does not grow without limit.

4. Stop overlapping runs with a lock

If a job takes 20 minutes and runs every 15, the second run starts while the first is still going. Two copies fight over the same files and database rows. A lock prevents that. On Linux, flock takes an exclusive lock on a file, and -n (non-blocking) makes the new run give up immediately if the lock is held:

*/15 * * * * flock -n /tmp/report.lock /opt/jobs/.venv/bin/python /opt/jobs/report.py >> /var/log/report.log 2>&1

While the first run holds /tmp/report.lock, the next one exits at once with a non-zero status and the job is not started twice. When the first run ends, or the process dies, the lock is released automatically, so a crash cannot leave the job stuck for good. That is the advantage over a hand-made "lock file" that you create and delete yourself.

5. Make the job safe to run again

Cron will run the job on schedule whether or not the last run worked. Design for that:

• Be idempotent: running the same job twice for the same period should leave the same result.
• Take the period as an argument instead of using "now", so you can re-run the previous day after a failure.
• Exit non-zero on failure, so a wrapper or monitor can see that it failed.
• Tell someone when it breaks. A job that silently stops running is the hardest failure to notice. Have the job ping a monitoring service when it finishes successfully, and alert if the ping stops arriving.

When to move on

Cron does one thing: start a command at a time. It does not know about dependencies between jobs, retry failures, backfill missed periods or show you a history. When you start needing those, or when you have more than a handful of jobs, a workflow orchestrator pays for its extra complexity. Until then, a clean crontab with locks and logs is hard to beat.