Keep cron monitoring in the same repo as the cron jobs

Someone adds a nightly export to the crontab. Six weeks later it stops running and nobody notices for four days, because the monitor for it was never created. A worse version: a monitor does exist, the job moved from 03:00 to 05:00 in a deploy, the monitor still expects 03:00, and it has been crying wolf every night since. Everyone muted it.

Both failures have the same root. The schedule lives in version control: a crontab, a Kubernetes CronJob, a Terraform aws_scheduler_schedule, a Helm value. The monitor lives in a browser tab. Two sources of truth for one fact, updated by different people at different times, and they slide apart.

Two kinds of drift

Missing. A job ships without a monitor. There is nothing to alert, so the job is invisible until someone downstream asks where the data went. The gap between “deployed a scheduled job” and “created its monitor” is a manual step, and manual steps get skipped under load.

Stale. The job's schedule, name, or command changed and the monitor did not. Now it checks the wrong window, or alerts on a cadence that is no longer real. A false alarm that fires every night gets muted within a week, and a muted monitor protects nothing.

Put the monitor next to the job

The fix is the one you already apply to infrastructure: define it as code, keep it in the repo, apply it on deploy. Three ways, cheapest first.

1. The ping is part of the job line. The job pings a URL as its last step, and that URL sits in the same file as the schedule. Change the crontab entry and you change the ping with it, because it is the same entry.

0 5 * * *  /opt/jobs/nightly-export.sh && curl -fsS -m 10 https://illari.dev/ping/$EXPORT_KEY

A wrapper does the same without touching the job's own code. The illari CLI reads its key from the environment, so the monitor binding lives beside the schedule in whatever renders the crontab or the unit file:

[Service]
Environment=ILLARI_KEY=%i-export
ExecStart=
ExecStart=/usr/bin/illari run -- /opt/jobs/nightly-export.sh

2. A reconcile step in the pipeline. After the deploy that ships the scheduler config, a script walks the jobs it just applied and upserts a monitor for each through an API. Adding a job to the manifest now creates its monitor on the next deploy.

# for every scheduled job in the manifest, ensure a monitor exists
jq -r '.jobs[] | "\(.name) \(.schedule) \(.timezone)"' jobs.json |
while read -r name schedule tz; do
  curl -fsS -X POST https://illari.dev/api/v1/monitors \
    -H "Authorization: Bearer $ILLARI_TOKEN" \
    -H "content-type: application/json" \
    -d "{\"name\":\"$name\",\"cronExpression\":\"$schedule\",\"timezone\":\"$tz\"}"
done

3. Infrastructure as code. One monitor resource per job, in the same Terraform module as the CronJob or the scheduler rule. terraform plan shows the monitor diff right next to the schedule diff, and a destroyed job takes its monitor with it.

What this buys you

A pull request that changes a schedule shows the monitor change in the same diff, so a reviewer catches “you moved this to 05:00 but the monitor still says 03:00.” Deleting a job deletes its monitor in the same commit, so there are no orphaned checks quietly passing forever. And the next engineer copies the pattern for the next job without needing to know which monitoring tool you use.

Where a dashboard still wins

A one-off backfill you will run three times does not need a Terraform resource. When you are still working out the right grace period and schedule, clicking is faster than a plan-and-apply loop. Codify it once it is stable and you expect it to stay.

illari is built for all three paths: the integration is a URL so the ping travels in the job line, there is a CLI wrapper, and a REST API with an OpenAPI spec for the reconcile step, which a Terraform provider could sit on top of.

The schedule is in version control. The check that the schedule is still running should be in there too, in the same file, changed in the same commit.