Kubernetes CronJobs: what kubectl won't tell you

kubectl get cronjobs shows a LAST SCHEDULE column ticking forward right on time. It looks healthy. All it actually confirms is that the CronJob controller created a Job object at the scheduled minute. Whether the Pod inside that Job started, ran to completion, or did anything useful is a separate question the CronJob resource does not answer at a glance.

The controller's job is narrow: watch the clock, create a Job on schedule. Everything downstream of that, scheduling the Pod, pulling the image, running the container, retrying on failure, is the regular Job and Pod machinery, and every quiet CronJob failure I have chased lived in that downstream half.

A busy queue skips the next run, not delays it

Set concurrencyPolicy: Forbid and the controller will not start a new Job while the previous one is still running. That is usually what you want, but if a run gets stuck (waiting on a pending Pod, blocked on a lock, a slow dependency) the next scheduled time arrives, finds a Job still in flight, and the run is skipped. No event fires that jumps out in a dashboard. You have to already be looking at kubectl describe cronjob to catch the skip, and even then it reads as routine. “job already active, skipping.”

concurrencyPolicy: Replace has the opposite failure: it kills the in-progress Job to start the new one, so a run that was three quarters done is thrown away and restarted, not resumed.

A missed run can be dropped, not queued

If the controller itself is down (a control plane upgrade, a restart) when a scheduled time passes, startingDeadlineSeconds decides what happens next. Leave it unset and Kubernetes tries to catch up on missed runs when the controller comes back. Set it, and once the deadline passes the missed run is simply not started at all, counted as a failed schedule with no Job and no Pod to inspect. Either behavior is reasonable, but it is a policy choice most teams never make on purpose, they inherit whatever the default was when the manifest was written.

A Job can “succeed” after failing twice

backoffLimit lets a Job retry a failing Pod before giving up. That is good resilience, and it also erases the signal. A Pod that got OOMKilled, retried, and succeeded on the third attempt shows up in kubectl get cronjobs exactly like a Pod that succeeded on the first try. The memory limit that is one bad day away from taking the job down for good never surfaces unless someone is watching Pod events, which nobody is, because the job “worked.”

The suspend flag has no expiry

spec.suspend: true is the normal way to pause a CronJob for a maintenance window. I have also left it set after the window closed more than once, which is a paused monitor nobody remembered to resume. The schedule still looks correctly configured. Nothing runs, and nothing complains, because a suspended CronJob is doing exactly what its spec says.

Make the Pod report on itself

Every one of these failure modes is invisible to a check that only asks the Kubernetes API “does this CronJob exist and does its schedule look right.” The fix is the same one that works outside Kubernetes. Have the workload call out when it finishes, independent of whatever kubectl says about the controller's own bookkeeping.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-export
spec:
  schedule: "17 3 * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: export
              image: your/image
              command: ["/bin/sh", "-c"]
              args:
                - |
                  ./export.sh
                  code=$?
                  curl -fsS -m 10 "https://illari.dev/ping/YOUR_KEY/$code"
                  exit $code

The ping happens inside the container, after the real work, so it reports the actual exit code, and it fires whether the Job succeeded on the first attempt or the third. A monitor with no fixed schedule (open-ended, alert if silent past a grace window) is often the better fit here over mirroring the cron expression exactly. startingDeadlineSeconds and retries already shift the real completion time around, and a silence-based check does not care by how much.

A CronJob's own status tells you the controller is doing its job. It does not tell you the workload did. The only signal that survives a stuck Pod, a dropped run, or a quietly suspended schedule is one the workload sends itself.