October 17, 2026 · 8 min read
Automated checks that never run: execute each one the day you write it
By Sahan, co-founder, systems and delivery
An automated check that has never been executed is not a check. It is a sentence that looks like one. The only way to know a check can fail is to run it once on the day it is written, against a case where the answer is known, and watch it say so. Skip that step and the check can sit in your system for months, reporting nothing, which everyone reads as “all clear”.
We found this out on our own operation in August and September 2026. A scheduled routine of ours reviews checks that other agents and people file against finished work. Day after day, most of the checks due that day could not answer their own question. None of them raised an error. They returned silence, and silence is what a healthy system also returns.
Why does a check that cannot run look the same as a clean result?
A check that cannot run returns an empty result, and an empty result is indistinguishable from “nothing found”. Query a table that does not exist and some tools return zero rows. Run a script with a missing argument and it prints a usage line to the wrong stream, so the next command in the pipeline sees no input and passes. The failure and the success have the same shape.
This is an old problem in monitoring, and the standard answer is to make the absence of a signal the signal. Healthchecks.io, which monitors scheduled jobs, is built on exactly that idea: it stays quiet while pings arrive on time and raises an alert when one is missing, so the silence itself is the signal that something went wrong (Healthchecks.io documentation, 2026). That protects you against a job that stopped. It does not protect you against a check that was never able to start, because nobody told the monitor that check should exist.
The same pattern shows up in the tooling that runs your gates. GitHub’s documentation on required status checks says that if path, branch or commit-message filtering skips a workflow, its required checks stay Pending and block the merge, while a job skipped by a condition reports success and does not block (GitHub Docs, troubleshooting required status checks, 2026). Two ways to not run, two opposite outcomes, one of them a green tick. If you have never watched your own check get skipped, you do not know which kind you have.
What did our own unrunnable checks look like?
They came in four shapes. All four were written by a capable person or agent in the middle of a task, none was run at the time, and each was caught only when the routine that reviews them tried to execute it later.
The check that only a human could close. Two checks were filed as work for an agent. Their own text said there was no data source to read and the answer had to be a manual pass or fail. An agent can never close that. It sat in the queue, due again every day, producing no verdict.
The check that named a column that does not exist. One prescribed a join between two tables on a field that neither table had. The database answered with an error. In a different tool that same query would have returned no rows, and no rows would have looked like a good outcome.
The check pointed at the wrong database. Several checks queried tables that were not in the project the author happened to have open. The feature lived in a different project. Against the wrong one, the query returns nothing forever, and the check passes for the wrong reason, which is worse than failing.
The check whose precondition was not scheduled. One tested an endpoint that would return “not found” until two other tasks landed. The check never said so. It simply could not be conclusive until someone else’s work finished.
There was a fifth, quieter variant. One check was correct, but the decision it was meant to inform had already been made and closed before the check came due. The check had no one waiting on it. Nothing was wrong with its query, and it still told us nothing.
What is the cost of rolling an unrunnable check forward?
Each time the review routine hit one of these, the tempting move was to mark it inconclusive and move the due date by a day. That feels like diligence. It produces another non-answer tomorrow, and the same one the day after. We ended up with checks that came due three days running and failed the same way each time, because their subject did not exist.
The rule we adopted is blunt: a check that cannot become conclusive tomorrow is not a check, it is a zombie. Close it with the finding written on it, and file the real question as a new piece of work with a named owner. A clause in the check saying “keep polling” is a licence to wait for state to change. It is never a licence to re-run a query whose subject does not exist.
We also learned to read the failure before reading the verdict. When a check returns nothing, the first question is not “is everything fine?” It is “does the thing I am asking about exist?” Does the table exist, does the agent have that name, is the flag in the list the script audits. Prove the name first, then trust the silence.
The same failure, one level down: a number that is never computed
The cleanest example of silence passing for health came from a report, not a check. A dry run of a data backfill printed a count of rows that would be replaced, and the count was zero every time. Zero is plausible, so people believed it, and it sent a full debugging cycle after components that were not broken.
The count was never computed. The totals rolled up a flag that only the live path sets, so every dry run reported zero regardless of what it found. The fix was to pull the rollup out into a small function a test could reach, because a counter you cannot test is where a constant hides. The tests were then confirmed to fail against the old code. That last step matters: a new test that passes on both the broken and the fixed version proves nothing.
Same family, different costume. A check that cannot fail and a number that cannot vary are the same defect.
What should you do this week?
Run each check once, today, against a case you can predict. The table below is the short version of what we do now.
| Question | What a real answer looks like | What a fake answer looks like |
|---|---|---|
| Does the thing it queries exist? | You see the name resolve before you run anything | Zero rows from a name nobody confirmed |
| Can the tool produce output at all? | Output appears when you deliberately give it a failing case | Empty output, which you read as “clean” |
| Can its owner actually close it? | An agent or script can reach the evidence and close it | The text says “manual pass/fail” on an agent’s queue |
| Is the preceding work in place? | Preconditions are listed and scheduled | The endpoint 404s until someone else ships |
| Is anyone waiting on the verdict? | A decision or a person is named | The decision closed last week |
| Can the number vary? | It changes when you change the input | It has read the same value on every run |
Then add two habits. First, make the check fail on purpose once, with a known bad input, and confirm it says so. A smoke detector gets tested by pressing the button, not by waiting for a fire.
Second, borrow the watchdog pattern from monitoring. The Prometheus operator’s runbooks describe a Watchdog alert that is designed to always fire, so that if it ever stops reaching the receiver you know the alerting chain itself has broken, and they suggest an external dead man’s switch to notify you when it stops arriving (Prometheus Operator runbooks, 2026). Apply the same idea to a review routine: have it report how many checks it actually executed, and alert on the count, not only on failures. A review that ran nothing and a review that found nothing must not look alike.
What is still open?
Plainly, not everything is fixed. The routine that reviews checks now closes unrunnable ones with a finding instead of rolling them forward, and the writing step asks the author to run the check once. That second part is a habit, not an enforced gate, and it depends on the author following it. We have not yet made “this check has been executed at least once” a field the system refuses to save without. Until we do, the safeguard is the reviewer catching the leftovers, which is the thing we are trying to move earlier.
One of the findings is also still waiting on a person on the data side to correct a wrong record before a repeat run can be trusted. We have the finding written down. The correction is not done.
For guardrails around what an agent may do in the first place, see AI agent guardrails for production. For the question that sits after the check passes, how you know the automation is still alive three days later, see how to know an automation is still working. This post is the step before both: making sure the check can run at all.
Where this fits
A check that has never run is one of the first things an AI operations audit looks for. We list the automated checks, gates and verifications you rely on, run each one against a known case, and report which ones can fail. The audit also leaves you with a short written rule for new checks. The same discipline runs across our own agent fleet, where a tomte, our word for one production agent with one defined job, is only trusted once the check that watches it has been seen to fail on purpose.
Start async
Send us a short list of the automated checks, alerts and tests you rely on and tell us which one you would be most surprised to learn had never run. We will reply within one business day with what we would test first. No call required.