How to know if your cron job silently failed

The worst cron failures aren't the loud ones. They're the jobs that quietly stopped running three weeks ago, and you find out when you reach for the thing they were supposed to produce.

Cron is deliberately minimal. It runs a command on a schedule and moves on. It doesn't track whether the command worked, whether it ran at all, or whether the machine was even awake when it should have. If something breaks, cron won't come find you.

Why cron jobs fail without anyone noticing

A scheduled job can go dark for a lot of ordinary reasons:

• The command started failing — a moved file, an expired credential, a full disk — but nothing was watching its exit code.
• The crontab itself got wiped or not restored during a server rebuild, image redeploy, or user migration.
• The box was down, suspended, or rebooting at the scheduled minute. Cron doesn't catch up on missed runs.
• cron tried to email you the output, but the host has no mail transfer agent configured, so the mail went nowhere.
• A dependency further up the pipeline stopped producing input, so the job runs and exits cleanly having done nothing.

Every one of these leaves no error in front of a human. The job is just absent, and absence is exactly the thing an error-reporting tool can't report on.

Start with the basics on the job itself

Before reaching for anything external, make the job honest about its own failures. For shell scripts, fail fast and fail loud:

#!/bin/bash
set -euo pipefail   # exit on error, undefined var, or failed pipe

pg_dump "$DATABASE_URL" | gzip > /backups/db-$(date +%F).sql.gz

set -e stops the script at the first failing command instead of charging ahead. set -o pipefail matters more than people expect: without it, pg_dump | gzip reports success as long as gzip succeeds, even if pg_dump died halfway through.

If your jobs run from cron directly, set MAILTO at the top of the crontab and confirm mail actually leaves the box. Piping output to a log you never read is not monitoring.

MAILTO="[email protected]"

0 3 * * *  /usr/local/bin/backup.sh

This helps with jobs that run and fail. It does nothing for jobs that don't run — no execution, no output, no email.

Catch the absence: the dead man's switch

The reliable way to detect a job that stopped running is to invert the problem. Instead of waiting for a failure signal, expect a success signal on a schedule, and alert when it doesn't arrive.

1. The job makes one HTTP request when it finishes successfully.
2. Something outside your infrastructure expects that request on a schedule.
3. If the request is late or missing, that external thing alerts you — because your job, by definition, can't.

In a crontab it's one appended command, guarded so it only fires when the real work succeeded:

0 3 * * *  /usr/local/bin/backup.sh && curl -fsS https://example.com/ping/your-key

The && means a failed backup.sh never reaches the ping, so a broken job looks identical to a job that didn't run at all — silent — which is precisely what you want to be alerted about. curl -fsS keeps curl quiet on success but loud (and non-zero) on an HTTP error, so a failed ping still shows up in the job's own logs.

What to use

You don't have to build the receiving end. For a handful of jobs on straightforward schedules, Healthchecks.io's free tier (open source, and self-hostable if you want) or Uptime Kuma if you'd rather run everything yourself will cover you.

Where a bare “alert if no ping for N minutes” check starts to hurt is real schedules: it produces false alarms around daylight-saving transitions and doesn't understand a cron expression at all. If your jobs run on specific cron schedules — especially across timezones — that's what illari is built for. Give it the cron expression and an IANA timezone and expected run times are computed with DST handled, so a 0 2 * * * job doesn't page you twice a year when the clocks move. The cron expression tester shows that math for any schedule; monitoring 25 jobs is free.

Whatever you use: the goal is that a job which stops running becomes something you find out in minutes, not something you discover when you need its output.