Key takeaways
- Every blog post on our site went down at once, including ones live for a month. The homepage was fine. The blog index was fine. Every article was dead.
- The cause was one corrupt post. The CMS could not build a list containing it, and our site reads every article through that list.
- Our own health check was hiding it, passing the CMS error along as a valid feed with a success code.
- The fix was a fallback copy of the feed built from a different endpoint. That copy needs its own daily refresh or it quietly ages out.
- Days later the refresh job failed for an unrelated reason and three new posts were unreachable for hours. A backup needs monitoring too.
The alert was not an alert. Somebody clicked a link to one of our articles and got an error page. Checked another one. Same. Checked one published back in July that had been fine for a month. Same.
Fifty three articles, all dead, all at once. Meanwhile the homepage loaded, the careers page loaded, the sitemap loaded, and the blog index loaded perfectly with all its post cards sitting there looking healthy.
The index lied
That last bit sent us the wrong way for a while. The index looked completely normal, every card and title and image, so the first read was that the blog was mostly fine with a few broken links.
Those cards are static markup. They are not built from live data, so the index renders beautifully whether the content behind it exists or not, and every link on it was pointing at a dead page. If you are checking whether a section of a site works, the listing page is not the test. Open an actual item.
One post took down all of them
Our site does not store articles. It reads them from the CMS through a feed, so that feed is the read path for the entire blog. When we asked for it directly, the CMS returned an error saying it could not generate a preview of the blog list. Not a timeout, not rate limiting. Consistent, every time, on every combination of settings we tried.
So we went looking for the article it choked on, asking for the list in small windows. Ten starting at forty, fine. Ten starting at fifty, error. Twenty starting at forty, error. Fifty starting at zero, fine.
That pattern points at exactly one place. There were fifty three published articles and the last one was corrupt. Any request whose window included it failed, and every window the feed generator uses includes the most recent posts. One damaged record, and every article on the site became unreachable.
Re saving that post in the CMS clears it. But we were not going to leave the whole blog depending on that being true.
Our own check was covering it up
We had a small endpoint on the site that fetched the feed and passed it through. It returned whatever came back, labelled as a feed, with a success code attached. Always.
So when the CMS returned an error message instead of a feed, our endpoint handed that error onward as though it were a perfectly good feed and reported success. It looked healthy, and it is why the first hour of investigating went into the wrong layer entirely. It now requires the response to actually contain an article before it will trust it. If you put something in front of another system to check on it, make it look at what came back. Otherwise it just repeats whatever it was told, in a more confident voice.
The fallback, and why it needs its own alarm clock
The fix that got the site back was a second copy of the feed, built a different way. The endpoint that returns one post at a time does not run the broken list logic, so we walked the posts individually, rebuilt the feed ourselves, and saved it as a static file the site falls back to when the live feed fails.
Fifty two of the fifty three came back immediately. The corrupt one returns a not found instead of taking everything else down with it, which is the correct blast radius.
Then the obvious problem. That file is a snapshot, not a mirror. Publish a new article and the fallback does not know about it. If the live feed is broken that day, your newest post is the one that does not exist.
So a job runs every morning, rebuilds the snapshot, and deploys it only if the content actually changed. That is the part people skip. Building the fallback feels like finishing. Keeping it current is the thing that makes it worth having.
Then the fallback failed too
Which brings us to a few days ago, and the reason this post exists rather than being a tidy story about a clever fix.
Three new articles were published. All three returned not found. The live feed had broken again with the same list error as before, which is exactly what the fallback exists for, but the snapshot job had also failed that morning. The site was still serving a snapshot from several days earlier that knew nothing about the three new posts.
Why it failed is almost funny. The job did all its work correctly, rebuilt the feed, every article parsed. Then at the very end it tries to trim its own log file and collides with its own logger still holding that file open. It crashes on the tidying up, after the work is done, and reports failure.
So the job that exists to cover for an outage was reporting failure for a reason unrelated to its actual work, and nobody read that report closely enough for it to matter. Two failures stacked, and neither on its own would have been visible.
Why this matters if you run a business here
Most sites we look at as a GHL consultant Bay Area shop have one of these underneath them. Pages get pulled from somewhere else the moment somebody visits. Normal and fine, right up to the day the somewhere else has a problem, and then your whole site goes with it. None of it your fault, all of it your problem.
What we would tell any owner to do about it:
- Know what your site depends on to render. If content comes from another system, that system is part of your site.
- Test a real page, not the listing. A perfect index proves nothing about the pages it links to.
- Never let a passthrough report success without checking the content. It will hide the outage it exists to reveal.
- Have a fallback copy, and give it a scheduled refresh. A stale backup fails exactly when you need it.
- Watch the thing that watches. Our backup job reported failure for days over a detail unrelated to its actual work.
Every article on our site is live again, and there are now two independent ways for them to render. That is not sophistication, it is just having been caught once.
If you want a site that does not disappear because someone else had a bad afternoon, come see how ours is put together at optechsol.llc.