October 14, 2026 · 6 min read
How to know an automation is still working after day one
By Anna, co-founder, build and content
You know an automation is still working when its newest output is recent enough, and you checked that yourself a few days after launch. Not the day it shipped. A day-one test proves the plumbing connects. It says nothing about whether the thing keeps running once the person who built it has moved on to the next task.
We learned this the slow way in October 2026, on our own systems, twice in one week. Both times we had told ourselves the work was done. Both times a check three days later said otherwise.
Why does an automation that worked on day one stop working?
Day-one success shows that every part connects. It does not show that anything keeps the parts alive. Jobs get unloaded at logout, checkouts drift from their source, a review step sits unmerged. The failure is quiet because nothing crashes: the job simply stops producing, and a stopped job sends no alert.
Gartner put a number on the wider problem in a June 2025 press release: it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing rising costs, unclear business value and inadequate risk controls (Gartner, 2025). Quiet decay is how many of those projects reach the point where someone asks what they are paying for.
The pattern is simple. A system that has to stay current is not finished when it first works. It is finished when it has stayed current for several days without anyone touching it.
What did we get wrong, in our own operation?
We reported two things as live that were not. Neither failed loudly. Both looked fine on the day.
The first was a memory system for our internal assistants. On launch day the hooks fired, the sync ran and a test session answered correctly. We called it done. Three days later the data was stale, because nothing was keeping the underlying checkout and the curated facts current. A verification step we had set up said “the consolidating pull request exists,” and it passed, while that pull request sat unmerged for two days.
The second was a nightly job on a Mac, run through launchd, the macOS service manager. We loaded it straight from a file inside a code repository. A status command that same day showed it loaded, and we reported it live. Within days it was gone: the status command said it could not find the service, no log had ever been written and the job had run zero times.
In both cases the evidence we used on day one was true and useless. The pull request did exist. The job was loaded, that day.
Why is “it exists” not the same as “it is fresh”?
An existence check asks whether a thing is there. A freshness check asks how old the newest result is. Those are different questions, and automation fails in the gap between them: a file, a row, a scheduled entry or an open pull request can all exist for weeks while the work they stand for has stopped.
Apple’s launchd documentation explains the second incident exactly. User agents are loaded from three directories, one of which is the per-user LaunchAgents folder in the user’s Library, and a per-user launchd loads them when that user logs in (Apple Developer documentation). A job loaded from anywhere else lives until the next logout or reboot. We had read that page before. We had not made the job match it.
The status label was never the problem. A label that says “loaded” or “merged” describes one moment. It cannot tell you about Thursday.
What does a freshness check actually look like?
A freshness check has three parts: a named output, a maximum age, and a person who reads the result on a date. If any of the three is missing it is a hope. Write the pass condition as a measurement of the output, never as a statement about the machinery.
| Weak check (passes on day one) | Freshness check (fails when it should) |
|---|---|
| A pull request for the update exists | That pull request is merged, and the live copy equals its source |
| The scheduled job shows as loaded | The job’s log file was written within its expected interval |
| The report was generated once | The newest report is younger than the schedule says it should be |
| The sync command exited with code zero | A record created at the source this morning is visible at the destination |
Pick the right-hand column for each automation you run, then put a date on the calendar two to three days after launch. That date is the check. On it, you look at the age of the newest output and write down the number.
We also ship a small health command with each system, one command that prints those ages and fails if any is past its limit. If a person has to assemble the answer from three dashboards on the day, they will skip it.
How do you catch a job that never ran?
A job that never ran produces no error to catch, so you watch for silence instead. The standard pattern is a heartbeat: the job reports in after each successful run, and a monitor alerts when the report does not arrive.
Healthchecks.io documents the mechanics. Each monitored job gets its own ping address, a check becomes late when its expected run time passes without a ping, and a grace time defines how long to wait before the check goes down and sends the alert (Healthchecks.io documentation). The point of the grace window is that a run a few minutes late is not an outage.
Two rules make a heartbeat honest. Send the ping only after the work succeeded, not when the job starts, or a job that crashes in the middle still looks healthy. And keep it separate from the logs: logs tell you what happened when the job ran, a heartbeat tells you when it did not.
The Google SRE book makes a related point about errors that never raise an error code. It separates explicit errors from implicit ones, such as a successful response carrying the wrong content (Google SRE book, chapter on monitoring distributed systems). A run that exits clean and writes nothing is the automation version of that. The heartbeat has to be tied to the output, or it only proves the process woke up.
How do you apply this in your own business this week?
Do four things this week, in this order, and nothing else.
- List every unattended automation you rely on: scheduled reports, syncs, content publishing, chasers. Most small businesses have more than they think.
- For each one, write down its output and the longest acceptable gap between outputs.
- Check the newest output’s age today. Anything older than its limit is already broken, and you have just found out for free.
- For anything launched recently, set the three-day date now and name who reads it.
Our own fleet gives the context. As of July 2026 we run 41 agents, and across a 13-day window they completed 5,450 runs with zero failures (agent fleet proof page, 2026). That zero is worth something only because failures are loud and every run is recorded, and because we keep checking the age of what each agent produced rather than the status it reports about itself. The incidents above came from the places where we skipped that.
If you want the monitoring half of this, monitoring AI agents in production covers the five signals to watch and who should read them. This post is the narrower habit underneath it: do not call something done until it has stayed fresh.
Who runs these checks for you?
This is the recurring work in our Fractional AI Officer service: every automation we ship or inherit gets a named output, a maximum age and a dated freshness check, and a weekly written brief says which ones passed. A tomte, our word for one production agent with one defined job, is only finished when its output stays current, and we write that into the handover rather than assume it.
Start async
Send us a short description of what you have running unattended and we will tell you, in writing, which of it we would check first. We reply within one business day. No call required.