Key takeaways
- Our whole automation fleet stopped for nineteen hours and every dashboard stayed green the entire time.
- Nothing was broken. Finished jobs left their processes running, those held every available slot, and new jobs were refused rather than failed.
- A refusal writes no error, no log line and no change anywhere. It looks exactly like a job that ran and found nothing to do.
- Spreading the schedule out does not help. The slots were held by dead work, not busy work.
- When an automation stops producing, the first question is not what is wrong with it. It is whether it ran at all.
One morning the approval inbox had nothing in it. No social posts staged, no blog drafts, no ad copy. Which was strange, because about a dozen jobs run on a schedule here.
Everything said it was working. Every job enabled, every one with a healthy next run time. No error anywhere, in any log, in any file.
The fleet had been dead for nineteen hours.
What was actually happening
The scheduler allows a small number of jobs at once. When the slots are full a new job is not failed, it is refused, and it tries again in a minute. Sensible behavior. The problem was what was holding the slots.
Twelve processes from earlier runs were still resident. Not stuck mid task, not crashed. They had finished their work, written their output, and then simply never exited. The scheduler cannot tell a process doing work from a process that forgot to close, so it counted all twelve as busy and refused everything else.
At the worst point we found sixty six of these dating back four days. One job had been refused over two thousand times, roughly once a minute since its first scheduled run. Across the fleet that was about thirteen thousand refusals in a day, every one of them silent.
Why nobody saw it
A refusal produces no error, because nothing ran to produce one. There is no failed run in any history, nothing written to any log, and no file anywhere that changed. From the outside it looks precisely like a job that ran, checked, and correctly decided there was nothing to do today.
The status screen made it worse. Every job read as enabled with a next run time coming up soon, which looks healthy at a glance and means nothing at all.
The honest signal turned out to be an absence. A job that has never run does not have a last run time set to empty. It has no last run time at all. So a check asking is the last run recent sails straight past a job that has never executed once, and nine of ours were in exactly that state.
The fix that was not a fix
Our first instinct was that too many jobs were firing at once, so we thinned the schedule. Halved a couple of cadences, staggered the morning ones an hour apart.
It changed nothing, and in hindsight it could not have. The slots were held by processes that finished days earlier, so there was no live demand to reduce. We watched four jobs scheduled at four different times all get refused in the same one second tick, hours after every one of their times had passed.
That is the trap. The obvious fix addresses the symptom you assumed, and when it does not work you decide the problem must be somewhere else entirely.
The actual fix, and the second failure mode
Since the finished runs had already done their work, killing them costs nothing. Their output was already staged. So a cleanup job now runs every five minutes and ends anything that has clearly overstayed. A scheduled job older than three hours is gone. An interactive session somebody might be sitting in front of gets twelve hours, because those hold no slot and killing one only interrupts a person.
Then we found the second way this happens, which is nastier. If an unattended job asks to do something it has not been pre approved for, it stops and waits for someone to approve it. At four in the morning nobody does. It waits forever, holding its slot, burning enough CPU that it does not even look idle. Three of those freeze everything. So the cleanup job has a second pass that spots a session waiting on an approval nobody is coming to answer, gives it ten minutes in case a human does, then ends it.
And one more, arguably the meanest. Sessions authenticate, and that authentication expires. When it does, every scheduled job exits within about eight seconds having done nothing at all. Our wrappers ended on a success line, so the operating system recorded each of those as a successful run, and two nightly builds produced empty output for days behind a row of green ticks. The wrappers now pass the real exit code through and refuse any run whose log comes back suspiciously small.
The lesson we keep relearning
When we first wrote up this fix, the notes said the cleanup job was installed and handling it. Four days later somebody checked and the job did not exist. The script had been written. The documentation said it was scheduled. It had never been registered. The fleet had starved again in the meantime, behind forty three processes.
So the rule now is that when a fix is a scheduled job now handles this, you go and confirm the job exists on the machine. Documentation claiming something runs is not the thing running.
Why this matters if you run a business here
Every business we set up with marketing automation San Jose side ends up depending on things that fire without anyone watching. Follow up sequences, review requests, reminders, reports. The failure to fear is not the one that sends the wrong message. It is the one that sends nothing and tells you it went fine.
What to put in place, none of which needs a developer:
- Ask when did this last produce something real, not is this switched on. Enabled is a setting, not evidence.
- Make silence loud. If a job that runs daily has produced nothing in two days, something should tell you.
- Check output at the far end, not status at the near end. Look at the post, the email, the record.
- Before you debug the logic, confirm it ran. Most of the time we thought a system was broken, it had never been invoked.
- Watch for green lights with nothing behind them. A success tick from a job that produced nothing is worse than an error.
None of that is glamorous. It is the difference between automation you can leave alone and automation you have to babysit, which is the only difference that matters after the first month.
If you want systems that tell you when they stop rather than quietly going quiet, see how ours are built at optechsol.llc.