Key takeaways
- Documentation does not stay true on its own. Ours drifted from what was actually running, twice, before we did anything about it.
- So a script runs every morning and checks 27 things about our own system against what our notes claim is true.
- One file is the source of truth. Every other place is checked against it, never the other way round.
- When it fails it opens exactly one task naming what broke, and refuses to open a second while that one is still open.
- Today it passed 22, failed 2 and warned on 3. Both failures were real drift nobody had noticed.
There is a specific kind of lie that builds up in every business running on software. Nobody tells it on purpose. It is the gap between how you describe your setup and how your setup actually behaves right now.
You wrote the process down in March. In April someone changed a setting. In May a tool updated itself. By August the document is confident, detailed, and wrong in three places, and nobody knows which three.
We hit this hard enough to build something about it. Our own notes claimed a piece of self healing was installed and running. It was not. It did not exist yet. That claim sat there for four days while we made decisions on top of it.
The fix was a test, not a better document
Rewriting the document would have bought us a few weeks. The document was not the problem. The problem was that nothing anywhere ever compared the document to reality.
So now a script runs at 7:30 every morning and does exactly that. It makes 27 checks across four layers of the business, and every check has to produce evidence, not an opinion.
Layer one is infrastructure. Is the server we host actually up, is its certificate valid and not about to expire, is the software pinned to the exact version we said we pinned it to.
Layer two is our team of AI assistants. We have nine of them, and each one is defined in four separate places: the command center, the agent files, the chat relay, and the portrait assets. Four definitions of the same nine characters is four chances to disagree.
Layer three is the phone. The AI agents answering our lines have their prompts stored in our own files, and the test checks the match in both directions. Every documented agent should exist, and every live agent should be documented.
Layer four is the wiring. Not is each thing alive, but are they connected. Do the buttons in our command center point at things that exist. Is the approval inbox present and readable.
The rule that makes it work
One file is the source of truth for who our assistants are. Everything else gets measured against it. Never the other way round.
That sounds obvious until you are in the middle of a mismatch at 8am. When two places disagree, the tempting move is to pick whichever looks more correct. That is how drift becomes permanent, because you just promoted the wrong copy. Deciding in advance which file wins means a mismatch has exactly one fix.
What happens when it fails
It opens one task on our internal board, naming the specific checks that failed. That is the whole response. No email blast, no alert channel, no dashboard nobody opens.
The part worth copying is what it does next. It will not open a second task while an identical one is still sitting there open. A failing check that nobody has got to yet would otherwise stack a new card every single day, and by day nine you have trained yourself to ignore that card.
Warnings never fail the run. They get written down and reported, but they do not raise anything. If everything is urgent then nothing is, and a test that cries wolf gets muted within a week.
Every run writes its evidence to a dated file, so you can see when something started failing instead of guessing. Those files get cleaned up after a month, because a year of daily reports is 730 files and nobody reads a haystack.
It runs alone, on purpose
This test is a plain script on its own schedule. It is deliberately not part of the automation fleet it watches.
We learned this the expensive way. Our automations run in a limited number of slots, and if the watchdog competed for those same slots, a bad day could starve the thing meant to tell you about the bad day. A monitor that the outage can take out is decoration.
The honest part
Today's run passed 22 checks, failed 2, and raised 3 warnings.
Both failures were real. One found stale duplicate copies of our AI assistants left behind on the relay, old identities that never got cleaned up when we rebuilt them. The other found that three of the nine were set to answer only the owner, which meant everyone else on the team was talking to them and getting silence.
Nobody had reported either. Nobody would have. Neither throws an error. They are the kind of thing where someone quietly stops using a tool because it never answers them, and never mentions it.
On its very first run this test caught drift that was already two days old.
What this means if you run a business in the Bay Area
Most small businesses we talk to around San Jose have the same problem in a smaller package. A Go High Level Bay Area setup with automations built over a year, a document describing how it all works, and nobody who can tell you today whether the two still agree.
Usually you find out from a customer. A form stops sending its notification and you hear about it because someone says they never got a reply.
You do not need 27 checks. Start with three:
- Write down what should be true. Which forms create which contacts, which automations send which messages, which numbers ring where. Boring, specific, one line each.
- Test one of them by hand every week. Submit your own form. Call your own number. Take the two minutes.
- Decide which record wins before you need to. When your notes and your account disagree, which one gets corrected. Pick now, while nothing is on fire.
Trust in a system should be earned every morning, not assumed because it worked in March.
If you want a setup where somebody is actually checking, that is the kind of thing we build. Have a look at optechsol.llc.