Exit code 0 but the backup was empty: catching cron jobs that lie

“It ran, exit code 0, no errors in the log.” And the backup file is 45 bytes. Running a job and doing what the job was for are two different things, and exit codes only tell you about the first one.

How a job succeeds and still fails

Some failure modes that leave a clean exit code behind:

• Empty database dump. Auth failed, pg_dump wrote an error to stderr and exited non-zero — but it was piped to gzip, which happily compressed the error message and exited 0.
• Partial sync. An rsync or API export pulled 200 of 10,000 records before the connection dropped, and the wrapper script treated “some data” as done.
• HTTP 200 with an error body. The upstream returned 200 OK with {"error":"rate limited"} in the body, and curl without -f is perfectly happy with that.
• Stale output. The job wrote yesterday's data again because the source it reads from stopped updating.
• Silenced errors. Somewhere in the script is a || true that was added to get past a flaky step and never removed.

Fix what you can inside the job

The first line of defense is making the job stricter about its own success.

#!/bin/bash
set -euo pipefail

DEST="/backups/db-$(date +%F).sql.gz"
pg_dump "$DATABASE_URL" | gzip > "$DEST"

# a real dump for this database is never under a megabyte
MIN_BYTES=1000000
actual=$(stat -c%s "$DEST")
if (( actual < MIN_BYTES )); then
  echo "backup looks truncated: $actual bytes" >&2
  exit 1
fi

set -o pipefail alone would have caught the classic pg_dump | gzip case. The size check catches the broader “technically produced a file” case. The same idea generalizes: after the work, assert something that's only true if the work actually happened —

• row count in the destination is within range of the source
• the newest record is from the last 24 hours
• the output file is larger than some floor and newer than the input
• a known-present key exists in the result set

…and exit 1 when it isn't. Now a wrong run is a failed run.

The problem with keeping it all in the script

In-script assertions work, but they have real costs. You edit every job to add them. The thresholds get hard-coded and drift out of date. And a script that exits 1 still relies on something noticing that exit code — which brings you back to the silent-failure problem: if the job doesn't run at all, there's no assertion and no exit code to catch.

What's missing is somewhere outside the job that knows what a healthy run looks like and can compare against history — not just “did it check in” but “did it check in with a plausible result.”

Report the result, not just the run

The heartbeat pattern — a ping when the job finishes — extends naturally here. Instead of an empty ping, the job sends a little structure about what it did:

rows=$(psql "$DATABASE_URL" -tAc "select count(*) from orders")
bytes=$(stat -c%s "$DEST")

curl -fsS "https://example.com/ping/your-key" \
  -d "exit_code=0" \
  -d "rows=$rows" \
  -d "bytes=$bytes"

Now the monitor can alert on more than absence: a run whose bytes dropped by an order of magnitude from its usual value, a rows count that went backwards, an exit_code that isn't 0, a run that took five times longer than its own baseline. The thresholds live next to the history instead of hard-coded in fifty scripts.

This is the line between plain heartbeat monitoring and structured run data. A few tools do it — Drumbeats and Cronping both have a version — and illari is one: the free tier answers “did it run on schedule,” and Pro ($20/month flat, unlimited monitors) adds exit codes, duration, output capture, and duration-anomaly alerts on the same one-line integration. If you want that check without standing up a second tool or hard-coding thresholds into every script, that's the case it's built for.

Exit code 0 means the process ended. Whether it did its job is a separate question — and one worth asking on every run.