Key takeaways
- A script writes one of our configuration files from scratch. Over time, people added three kinds of notes to that file by hand, and the script had never heard of any of them.
- So our own documented instruction, regenerate the file with the script, deleted 105 hand-written annotations every single time somebody followed it.
- Nothing errored. A regenerated file looks perfectly correct. It surfaced only because a two line addition produced a diff of 342 changed lines.
- The damage would have been silent and expensive: those annotations decide what each edition of a product installs, so the solo edition would have quietly installed the agency setup.
- There are two honest fixes. Teach the generator to carry what it does not understand, or make it refuse to overwrite what it did not write. We did the first, and proved it by running it twice and comparing hashes.
One of the builds we package is a CRM setup that ships in two editions, one for an agency with a team and one for a single producer working alone. Behind it sits a registry file listing every custom value the account needs: what it is called, what tier it belongs to, and what it is for.
Nobody types that file. A script writes it, because there are over a hundred values and a script does not make typos. The instruction in our own notes was simple. Need to change the registry? Edit the script and regenerate.
That instruction was a trap, and it had been one for two days before anybody noticed.
The file had grown things the script never knew about
After the generator was written, the file picked up three additions that were typed by hand, straight into the output.
The biggest was an edition tag. Twelve groups and 89 individual values got marked as belonging to both editions, the agency edition only, or the solo edition only. That was a real set of decisions, made by a person, value by value. Three groups also got a short written note about how they apply to the solo edition. And the top of the file got a block describing the editions in plain English, plus counts of how many values carried each tag.
None of that lived in the script. It lived only in the file the script writes.
So every time somebody followed the documented step, the generator rebuilt the file from what it knew and wrote it over the top. All 105 annotations gone. No warning, no error, and a result that looked exactly like a clean, correct registry, because apart from the missing annotations it was one.
How it finally showed up
We were adding two values. A two line change. We regenerated, as instructed, and looked at the diff before committing.
The diff said 342 lines had changed.
That is the only reason this was found by us and not by a customer. A two line edit cannot legitimately touch 342 lines,. If the change had been fifty lines, the damage might have hidden inside it.
Why this was dangerous rather than annoying
Losing notes is annoying. This was worse, because those notes were not decoration. They were load bearing.
The script that installs an edition into a new account reads the edition tags to decide which values to create. With the tags stripped, the registry looks completely untagged, and an untagged registry does not fail. It just installs everything.
So a solo producer would have received the agency edition's values in their account. No error on install, no failed step. The only place it would have shown is inside an account nobody at our end looks at day to day.
That is the class of bug worth fearing most. Not the one that crashes. The one that succeeds at the wrong thing.
The two ways out
A generated file that people have since annotated is not purely generated any more. Most configuration files end up there, because the output is where people look, so it is where people write.
The first is to teach the generator to carry what it does not understand. Before it writes, it reads the file it is about to overwrite, and anything hand-added gets carried forward onto the new version.
The second is to make the generator refuse. If the existing file contains anything the script did not put there, it stops and says so, and a person decides.
We chose the first here, for a practical reason. Moving a hundred judgement calls into the script would have meant transcribing them by hand. Carrying them was safer than copying them.
What the fix actually does
Four things.
It matches on a stable key. Every value in the registry has a key that never changes, even if its display name does. The carried annotations attach to that key, not to a position in the file or a label a person might tidy up later.
It recounts. The edition counts at the top of the file are rebuilt from the tags actually present, rather than copied forward from last time. A carried count would be a number that was true once.
It tells you what is new. Any value with no edition tag gets printed at the end of the run. A brand new value correctly has no tag yet, and now it shows up as a line somebody has to act on instead of a silent default.
And it was verified by behaviour, not by reading the code. We ran it twice in a row and compared the hashes of the two outputs. Identical.
What this means for your business
You probably have a file like this, even if you have never written a script. An export somebody added a notes column to. A monthly report rebuilt over the copy where someone highlighted the rows that matter.
The rule we took from it is short. Before you rebuild anything, compare the new version with the old one. If the difference is bigger than the change you made, stop, because something you could not see just disappeared.
For any Go High Level San Jose business running packaged builds, templates or snapshots, this is also a design question worth asking before the second person touches the file. Where do the human decisions live? If the honest answer is only in the output, the next regeneration is going to delete them.
Want this built for you
We build CRM systems and automations that survive the second person editing them, plus the websites and funnels in front of them. Start at optechsol.llc.