Watching dbt models without a data-observability platform

The nightly dbt build is green. A finance number is wrong anyway, because the test that would have caught it is set to severity: warn and the warning scrolled past in a log nobody reads. The run before that one did not happen at all, and the dashboard looked fine because it was showing two-day-old data.

dbt tests and source freshness are a real answer to data quality, and for a lot of projects they are enough. The gap is not in what dbt can assert. It is that dbt only checks when it runs, only checks what you wrote, and, on dbt Core, tells no one when the answer is bad.

What dbt checks, and when

The four generic tests (not_null, unique, accepted_values, relationships) plus singular tests (a .sql file that returns failing rows) and packages like dbt_utils and dbt-expectations cover most of what you would assert by hand. dbt build runs models and their tests interleaved, so a failing test stops its downstream models from being built on bad data. dbt source freshness compares a loaded_at_field against warn_after and error_after and writes the result to target/sources.json.

All of that fires only during a dbt invocation. Between runs, and for any column you did not write a test for, nothing is watching.

warn is not error, and some tests pass on nothing

A test configured severity: warn does not fail the build. That is the right setting for a check you are still tuning, and the wrong setting to forget about, because the run stays green while the warning count climbs. Running with dbt build --warn-error promotes every warning to an error, which is blunt but honest.

The other quiet case is a test that passes because there is nothing to check. An accepted_values test on a column that came back empty passes. A relationships test passes when the child table loaded zero rows. The assertion is green and the model is still wrong. Setting store_failures: true writes failing rows to a table you can inspect later, which helps once you already know to look.

The run you watch may not be the run that matters

In CI, dbt build --select state:modified+ on a pull request is green. That is a different invocation from the scheduled production build: a different target, the full graph, real data volumes, and a cron or orchestrator wrapped around it. On dbt Cloud that scheduled job has notifications. On dbt Core it has whatever you wired up, which is often nothing.

And the production build has a failure mode CI does not: it can skip. The cron entry was dropped in a deploy, the container hit its memory limit partway through the graph, the orchestrator is paused. There is norun_results.json to parse because dbt never started.

Send the test summary to a heartbeat

After the scheduled build, read target/run_results.json for the failure count and report it, with the exit code, to a monitor that lives outside your warehouse and your orchestrator:

#!/bin/bash
set -uo pipefail

dbt build --target prod
code=$?

failed=$(jq '[.results[] | select(.status == "fail" or .status == "error")] | length' \
  target/run_results.json 2>/dev/null || echo -1)

warned=$(jq '[.results[] | select(.status == "warn")] | length' \
  target/run_results.json 2>/dev/null || echo -1)

curl -fsS -m 10 \
  -d "failed_tests=$failed" \
  -d "warned_tests=$warned" \
  "https://illari.dev/ping/YOUR_KEY/$code"

Add dbt source freshness and post max_loaded_at_time_ago_in_s from target/sources.json the same way if stale sources are a concern.

Now the checks you already wrote in dbt reach someone. A missing heartbeat on schedule means the build did not run. A heartbeat reporting failed_tests above zero means it ran and something you assert is broken. On illari that second one is a metric rule (failed_tests is greater than 0); the alert names the value, and the failed and warned counts show on every check-in, so a slow rise in warnings is visible before it becomes a failure.

Where a platform still earns it

This works when you can enumerate the handful of things that must be true: these columns are never null, this source is never a day stale, this build runs every morning. When you need volume and freshness anomaly detection across every table without writing an assertion for each one, column-level lineage to trace a bad number to its source, or incident routing and history, that is what the observability platforms are built for.

dbt already knows when a model is wrong. The missing piece is usually just getting that verdict out of the run log and in front of a person the same morning.