Key takeaways
- A dozen automations we described as running daily had quietly stopped. Nobody noticed for three weeks, because the dashboards were green and the documentation was prose.
- The fix was not better monitoring code. It was a declared list of what is supposed to be running, in a file, that a machine can check.
- The alarm that checks it never fixes anything. Switching a job back on that somebody turned off deliberately is worse than leaving it off.
- It raises exactly one task, with a fixed title, so a two week outage does not become fourteen tickets.
- It is deliberately built out of the dumbest possible technology, because the thing it watches is the thing that would otherwise report the problem.
We run a lot of scheduled automations. Content gets built overnight, queues get filled, leads get researched, reports get assembled. For a long time the way we knew all of that was running was that a paragraph in our internal documentation said it was.
That paragraph said twenty one jobs were registered. An audit found sixteen, eight of them switched on, and about a dozen automations named in the documentation with no scheduler entry at all. Their state files had stopped moving weeks earlier. Everything looked fine the entire time.
Two ways an automation dies without telling you
The audit turned up two separate silent deaths in the same week, and they are worth describing because neither one looks like a failure.
The first was an application update that wiped the scheduler's registry. The jobs did not error. They simply stopped existing. Nothing failed, so nothing reported a failure, and the only visible symptom was output that stopped appearing in files nobody opens daily.
The second was subtler. Our overnight builds run through an AI runner, and that runner hit a monthly spend limit. Every nightly build then exited in about eight seconds having done nothing, writing a three hundred byte log. Meanwhile the jobs that post the already approved content kept running perfectly, against an empty queue. Green lights everywhere, two days of nothing built.
Same shape both times. The thing that would have reported the problem was the thing that broke.
The fix was a list, not a monitor
The instinct is to build better monitoring. We did something smaller first, and it did more work.
We wrote down what is supposed to be running, in a file, in three buckets. Expected, meaning it must be registered and switched on. Retired, meaning it is deliberately off and should never raise an alarm. And undecided, meaning it exists but nobody has made the call yet.
That file is now the declared truth, and the rule that comes with it is the part that makes it work: you edit the list in the same change that builds or retires an automation. If that discipline slips, the list rots and you are back to prose.
Why does a list beat a paragraph? Because a paragraph cannot notice anything. A sentence claiming twenty one jobs are running is just as confident when the real number is eight. A list has entries, and entries can be compared against reality by something that does not get tired or assume.
Five checks, and permission to fix nothing
Every morning at quarter to eight, a script compares that list to what is actually there. It checks the scheduler registry, the operating system's own task list including whether the last run exited cleanly, the runner logs for the stub sized ones that mean an authentication or budget failure, whether there is anything queued for tomorrow in each content lane, and finally whether there are jobs sitting on disk the list has never classified.
All five checks are read only. It never re enables anything, and that restraint is the most deliberate decision in the whole thing. Half our registered jobs are switched off on purpose. An alarm helpful enough to turn them back on would undo a human decision every morning, and we would learn to ignore it inside a week.
When it finds something, it opens one task on our board and stops. The title is fixed and carries no date and no count, on purpose, because that is what lets it recognise its own open ticket tomorrow and stay quiet instead of stacking up. One task per outage, not one per morning.
Built out of the dumbest thing that works
Two implementation choices matter more than they look.
It contains no AI at all. It is plain scripting, no models, no calls out to anything clever. That is not modesty. The runner being dead, out of budget, or wedged is precisely what the alarm exists to catch, so it cannot be built on top of the runner.
And it runs on the operating system's own scheduler rather than inside the same queue as the automations it watches. Our runner allows a limited number of jobs at once. A watchdog sitting in that queue would consume one of the slots and could starve the fleet it is supposed to be protecting.
Both are the same principle. Your alarm must not share a failure mode with the thing it monitors. It is obvious written down, and it is very easy to violate by accident when you are building fast.
What this means for your business
Every business running automations eventually runs some that have quietly stopped. It is not a question of discipline. Automation is invisible when it works and equally invisible when it does not, and a dashboard showing green usually means the reporting still works, which is a different claim.
Three things you can do this month, none of which need a developer. Write down what is supposed to be running, in a real list rather than in a document that describes it. Have one check per day compare that list to what is actually there, and let it tell one person. And decide in advance that the check reports rather than repairs, so you never wake up to find something helpfully switched back on that you turned off for a reason.
This is the boring end of what a GHL consultant Bay Area businesses hire actually does. Not the automation everyone gets excited about, but the small piece that tells you when it stopped.
Want this built for you
We build automation for small businesses, and we build the thing that tells you when it breaks, because one without the other is a slow motion problem. Start at optechsol.llc.