Your scheduled AI agent can finish clean and still be wrong

The nightly dependency-bump agent ran. Exit code 0, and a tidy summary in Slack: “Reviewed 12 packages, updated 4, tests pass.” It updated zero. It hit its turn limit while writing the summary, and the summary is what the model thought a good run would have looked like. Nobody checked, because nothing was red.

A scheduled agent is a job, and it fails the way pipelines fail: quietly, with a clean exit. The difference is that an agent will also narrate a success it did not have. The exit code and the summary are both unreliable, so the thing worth watching is what the run actually did.

Exit 0 is the harness, not the work

An agent framework runs a loop: call the model, run the tools it asks for, feed back the results, repeat until a stop condition. That loop terminates for several reasons, and most of them return cleanly. A max-turns cap is reached. A recursion limit trips. The context window fills. The model emits something the parser reads as “done.” From outside the process, a run that solved the task and a run that gave up at iteration 25 look identical: exit 0, some logs, a final message that reads like a result.

Your scheduler sees a clean run. Your log aggregator sees no errors. The signal that the run was hollow is in the numbers it produced, and by default nothing collects them.

Have the run report on itself

Make the agent, or the wrapper around it, write a short summary of what happened: why it stopped, how many turns it took, what it cost, and one or two counts that describe the actual job.

python agent.py --report run.json
code=$?

# run.json, written by the agent:
# { "exit_reason": "completed" | "max_turns" | "error",
#   "turns": 8, "tool_calls": 14, "cost_usd": 0.62,
#   "packages_updated": 4 }

curl -fsS -m 10 \
  -d "code=$code" \
  -d "exit_reason_ok=$(jq '.exit_reason == "completed"' run.json)" \
  -d "turns=$(jq .turns run.json)" \
  -d "tool_calls=$(jq .tool_calls run.json)" \
  -d "cost_usd=$(jq .cost_usd run.json)" \
  -d "packages_updated=$(jq .packages_updated run.json)" \
  "https://illari.dev/ping/YOUR_KEY/$code"

The jq boolean expression sends 1 or 0, so exit_reason_ok becomes a number you can threshold like the rest.

Threshold the numbers

Four cheap checks cover most of the hollow-run cases, and they are all thresholds on values the run just reported:

It did not run. No check-in on schedule. The agent host, the queue, or the cron entry failed, and there is no summary to judge because nothing started.

It gave up. exit_reason_ok < 1, or turns at the cap you configured. The loop ended on a limit, not on finishing.

It did nothing. packages_updated < 1 on a run whose whole point is to update packages. Swap in the count that matters: rows labelled, files changed, tickets triaged.

It overspent. cost_usd > 5 or tool_calls > 100 for a job that normally costs cents. Usually a retry loop against a broken dependency.

On illari these are metric rules on the monitor: cost_usd is greater than 5, packages_updated is less than 1, turns is greater than 20. A missing check-in is the first case on its own. Every reported number shows on the check-in, so the normal band for cost and turn count is visible before a rule fires.

What the numbers do not tell you

None of this judges the quality of the work. A run that updated four packages, took eight turns, and cost 60 cents can still have bumped a dependency past a breaking change and written a test that asserts the bug. Thresholds catch the gross failures for the price of a curl: the loop that spun, the run that spent forty dollars, the job that touched nothing. Correctness still needs an assertion in the pipeline, an eval suite, or a human reading the diff. The metrics are the floor, not the review.

An agent that exits 0 has told you the loop ended, nothing more. Have the run report what it did, and alert on the runs that did too little, too much, or nothing at all.