Key takeaways
- Our nightly build rendered a full day of social posts, media and captions, in its first thirteen minutes. Then it produced nothing for nearly three hours and was killed by its own time limit.
- Everything it made was finished and sitting on disk. None of it went out, because the step that writes the posting schedule never ran.
- The step in front of it was an optional video upload with no timeout of any kind. A stalled connection held the only output that mattered hostage.
- The run log was empty, which looks like an early crash and was not. The tool only writes its log at the end, so a killed run always logs nothing.
- Two rules came out of it. Order steps by what it costs to lose them, not by how the story reads. And any unattended network call needs a timeout, because nothing is watching and the only symptom is silence.
Our social posting runs on a build that happens overnight. It picks tomorrow's slots, renders the video or image for each one, writes the captions, uploads anything that needs uploading, and then writes the queue that the daytime job reads to actually post things.
One morning we had nothing scheduled. No error anywhere, no alert, nothing in the log. Just an empty day.
What made it strange is that the work was all there. Nine finished posts, media and captions, sitting in a folder. The build had done its job and then thrown it away.
Reconstructing a run that left no log
The log file was zero bytes, which reads like the build died immediately. That was the first wrong turn.
The tool we run unattended writes its log when it finishes. A run that gets killed never finishes, so it never writes. An empty log is not evidence of an early failure. It is evidence of no clean ending, which is a different thing entirely and points somewhere else.
What was left was the files. Every rendered asset carries the time it was written, and that gave us a timeline the log could not. The first asset landed a couple of minutes in. The last one landed thirteen minutes in. Then nothing, for two hours and forty-seven minutes, until the scheduled task hit its three hour limit and force-killed the process at exactly the second it was set to.
So the build did not crash and it did not run slowly. It stopped dead at a specific moment and sat there until something else ended it. That shape only comes from one thing: a call that is waiting on something and has no instruction about when to give up.
The step order was the bug
The order was: render everything, upload the short videos to YouTube as unlisted, then write the queue.
Read as a story that order makes sense. Make the thing, put it where it goes, write down the plan. Read by what each step costs you if it fails, it is exactly backwards.
The queue write is cheap, local, instant, and the only output that matters. Without it nothing posts. The upload is a nice-to-have: a Short can go up next week from files that already exist, and nobody notices.
We had put the fragile optional thing in front of the cheap critical thing, and the fragile thing could not give up. The upload script had no timeout at all. No overall limit, no socket timeout, nothing. A stalled upload will wait forever, because forever is what you asked for when you specified nothing.
The two fixes, and the one that is easy to get backwards
The first fix is ordering. Cheap, local and critical goes first. Optional and networked goes last, explicitly allowed to be skipped. Our build now writes the day's schedule before it uploads anything, and a skipped upload is logged as a deliberate skip rather than a failure. It happens regularly and costs nothing: the finished videos are still on disk.
The second fix is a timeout on the upload, and it has two properties that are easy to get backwards.
It has to fire when the process is stuck. That part is obvious. It also must not delay a normal fast upload, which is the part people get wrong, because a naive timer keeps the program alive until the timer expires even when the work finished in twenty seconds. A fifteen minute safety net that makes every successful upload take fifteen minutes is worse than no safety net at all.
Ours now has an outer deadline that defaults to fifteen minutes and a shorter guard on the connection itself, because a live upload always moves within a couple of minutes and one that has gone quiet is not coming back. Both of those were tested for both properties: that they fire when something is stuck, and that a normal upload still exits the moment it is done.
Check the folder before you rebuild
One more habit came out of that morning. A build that dies mid-run leaves finished work behind.
The instinct is to rerun the whole thing. That burns hours and, for anything that costs money to generate, real money. In our case the salvage was writing the queue and the approval rows from assets that already existed. Nothing needed regenerating. Every asset had been rendered before the build hung.
What this means for your business
The general version of this has nothing to do with video. Anything you run unattended has an order, and most orders were chosen by how the process reads rather than by what happens if a step fails.
Ask one question about each step: if this hangs, what do we lose? The step that costs the most should not be sitting behind the step most likely to fail. A nightly report that emails itself should write the report first and send second. An import that notifies a customer should record the record first.
Then ask whether anything in that chain talks to somebody else's server. If it does, and nobody is watching it, it needs a limit. A call with no timeout is not a call that never fails. It is a call whose only failure mode is silence.
If you are a marketing automation San Jose business running anything overnight, check two things this week: that the important output is written before the optional one, and that every external call has a deadline.
Want this built for you
We build automations that are ordered around what they cannot afford to lose. Start at optechsol.llc.