The pipeline that got slower every week

The nightly aggregation used to finish before you got in. Now it is still running at standup. The logs are clean, the exit code is 0, and it has been 0 every night for months. Nothing broke. It just got slower a percent or two at a time until forty minutes became normal.

Gradual slowdown is the failure that monitoring usually misses, because every individual run looks like the one before it. No error is thrown. No threshold is crossed on any given night. The job that took 4 minutes in January and 40 in April never had a bad run, it had ninety slightly worse ones.

Why the creep hides

cron does not care how long a command takes, only whether you asked it to start again before the last one finished. Exit codes describe the final state, not the cost of reaching it. And wall-clock time is almost never written down anywhere durable: it scrolls past in a log, or it lives in the scheduler UI for two weeks and then rotates out.

The causes are ordinary. A table that grows every day while the query still does a full scan because nobody added the incremental predicate. An index that was dropped in a migration and never recreated. A loop that issues one query per row, fine at a thousand rows and fatal at a million. A nightly VACUUM that silently stopped. A dependency that got slower, absorbed by a retry with backoff that turns one timeout into three attempts.

Record the duration somewhere a trend is visible

The first step is to measure each run and send the number somewhere that keeps a series. In bash, the SECONDS variable is a zero-effort wall clock:

#!/bin/bash
set -uo pipefail

SECONDS=0
./aggregate.sh
code=$?
elapsed=$SECONDS

curl -fsS -m 10 \
  -d "duration_s=$elapsed" \
  "https://illari.dev/ping/YOUR_KEY/$code"

If the job is a single command, the illari CLI does the capture for you: illari run --key KEY -- ./aggregate.sh sends the start ping, runs the command, and sends the completion with the elapsed time and the exit code.

Appending date +%s deltas to a CSV and running awk over it once a week is better than nothing. A monitor that plots the series and keeps the history is the point, because the signal is the shape of the line, not any single value.

Alerting on “slower than usual”

There are two useful checks and they catch different things. A fixed ceiling (duration_s > 1800) is trivial to set and catches the cliff: the night the job doubles. It does nothing for the slow ramp, because you set the ceiling with headroom and the ramp eats the headroom quietly.

A baseline comparison catches the ramp. Compare each run against the recent median and flag the ones that are well outside it. This needs a few runs of history to establish normal, and it only makes sense to flag slower, never faster. illari does this for run duration on its own: send a /start ping when the job begins and a completion ping when it ends, and it records the wall-clock time, compares against the recent baseline, and alerts on a completion that is far outside it. The duration shows on every check-in, so the trend is visible before the alert. The same baseline check runs on any number you report, so a rows_written or cost_usd that drifts off its norm alerts the same way.

The overlap trap

Left alone, a job that grows past its interval starts before the previous copy has finished. Two runs then hit the same tables at once, contend on locks, and double the load, so both get slower and the ramp steepens. Wrapping the command in flock prevents the overlap:

*/30 * * * *  flock -n /tmp/aggregation.lock ./run-aggregation.sh

That trades a pile-up for a skipped run, which is the better problem but still a problem you want to hear about. A /start with no completion is the signal: the run is either still going or was skipped because the last one had the lock.

What the number will not tell you

Duration is a symptom. It tells you when something changed, not what, so pair the alert with the next obvious checks: the query plan, the row counts, any migration that shipped that week. Postgres users can turn on auto_explain to catch the plan that flipped. And some slowdown is real. A dataset that genuinely grew earns a longer runtime, and the right response is to re-baseline, not to keep muting the alert.

A job that quietly tripled in runtime is telling you something changed upstream. Record the duration every night so the line has a shape, and the change shows up while it is still small.