Key takeaways
- One screen in our internal dashboard went blank. The page behind it answered every request with a success code and no data at all.
- The first request after every restart worked perfectly. Only the second and later ones came back empty, which is why every check anybody ran by hand said it was healthy.
- The route was fine, the permissions were fine, the files it reads were fine. The fault was one line of caching.
- The line stored the work onto the record as it was a moment earlier, and the work had already replaced that record with a fresh one. The next line read the new one, which was empty.
- It only bites when the work finishes without waiting for anything, which is exactly what a cached answer or a local file read does.
We run an internal dashboard that pulls together the things a morning starts with. One of its screens ranks what we should film that day. On September 6 that screen came up empty, and it stayed empty.
The obvious suspects were all clean. The address it calls answered, with a success code rather than an error. The files it reads were where they should be with sensible contents. Nothing in the logs, because as far as the system was concerned nothing had gone wrong. It was returning a successful response containing nothing.
That combination is worse than an error. An error names a place to look. A successful empty answer tells you the pipe works and quietly suggests the problem is your data.
The clue was in how we were testing it
Every time we restarted the service and checked, it worked. Real data, full list, exactly right. Then the screen would be blank again.
That sounds like something going wrong shortly after startup, which sent us looking at timers and file watchers. It was not that. The first request after a restart was doing different work from every one after it.
The screen is built from a scan that costs real time, so the answer gets held for a while and reused. The first request after a restart has nothing to reuse, so it runs the scan, waits for it, and gets real data. Every request after that finds a recent answer and takes the fast path, which does not wait for anything at all.
The fast path was the broken one. And the fast path is the one no human ever tests, because you restart, you check, it works, you move on.
One line, and it reads as correct
Here is the shape of it, without the code. There is a record holding the cached answer and a note of any work currently in progress. A request that finds work already running waits for that instead of starting its own, which stops ten open tabs kicking off ten identical scans.
So the code starts the work, writes it onto that record so other requests can find it, and then waits for it.
The trap is that the work itself, when it finishes, replaces the whole record with a fresh one holding the new answer. And when the work has nothing to wait for, it runs straight through to the end before the line that was going to file it has landed.
So the note gets written onto the old record, which by then nobody is looking at. The very next line reads the note off the new record, finds none, and waits for nothing. Waiting for nothing returns nothing, and the request answers successfully with it.
Every step in that sequence is individually reasonable. The fault lives only in the order the two halves happen, and that order depends on whether the work had to wait for anything, which depends on whether there was a cached answer, which depends on whether you just restarted.
The fix, and the twin we left alone
The fix is boring. Keep hold of the work directly, wait on that, and never look it up again through a record something else is allowed to replace. Three lines moved, no change to what the endpoint does.
The interesting part is where the pattern came from. It was copied from another screen in the same dashboard that has run fine for months, and that one is safe purely by accident. It always calls an outside service first, so its work always has to wait and the ordering never flips. Identical shape, one call away from the same fault, and reading it tells you nothing about which of the two you have.
We left it alone and wrote a comment on it. It is not broken, and rewriting working code on a hunch carries its own risk, but the next person copying from it should know they are copying a coin toss.
The habit that would have caught it
Call it twice. That is the whole lesson, and we paid a morning for it.
A cache has two behaviours by definition, the cold one and the warm one, and a single check only ever exercises one. Ours passed cold every time and failed warm every time, and nobody ran a warm check until we stopped blaming the screen. The other half is not trusting a success code to mean anything, because a page answering with nothing is still answering.
What this means for your business
You probably do not maintain a dashboard, but you almost certainly own something with a first run and a repeat run, and those are not the same run.
A welcome sequence behaves differently for a contact who already exists. A form works on a clean browser and misbehaves for somebody who filled it in last month. A quoting tool gives the right answer first and a stale one second. In every case the version you test is the fresh one, because testing means starting clean, and your customers almost never are.
Three things worth doing this week. Test anything automated twice in a row with the same input, and compare what came back both times. When something reports success, check that what arrived has content in it rather than treating the green light as the answer. And if a system is only ever right just after somebody restarted or cleared something, treat that as the clue it is, not as a fix.
For a GHL consultant Bay Area business, the version of this that costs real money is a form or a workflow that works for new contacts and silently does less for returning ones. Send yourself through it twice and read both results.
Want this built for you
We build internal tools and CRM automations, and we test them the way a customer would meet them rather than the way a developer would. Start at optechsol.llc.