Key takeaways
- We added a new page to our website, added it to the sitemap file, deployed, and the live sitemap still did not list it.
- The file we edited was not the file being served. A small piece of code builds the live sitemap from its own list, and the static file is only used when that code cannot load the blog feed.
- That second list had quietly drifted too. Two pages that were in the file were missing from what search engines actually saw, for weeks.
- The two lists exist for a good reason. Nobody wrote that reason next to the file people edit.
- While checking the facts for this post, we found it had happened again: one page added in September is in the file and missing from the live sitemap.
A sitemap is the list of pages you hand to Google so it knows what to crawl. Most business owners never look at theirs. It gets set up once, and after that you add a line when you add a page.
That is what we did in August when a new service page went live on our site. New page, new line in the sitemap file, deploy. Then we opened the live sitemap to confirm, and the new page was not there.
The obvious suspect is the deploy. Maybe it did not finish, maybe the old file was cached. We checked. The deploy was fine, and the file on the server had the new line in it. The sitemap a visitor got at that address still did not.
Two lists with the same name
Our site is a static site with a few small functions running beside it. One of those functions answers the sitemap address. When a request comes in for the sitemap, that function runs first, and the static file of the same name never gets a look in.
The function exists for a reason that is actually good. Our blog lives in the CRM, not in the site's code, and new posts go live there several times a day. A static file cannot know about a post that was published an hour ago. So the function pulls the blog feed, lists every live post, and puts the site's own pages on top from a list written into the code.
That list in the code is the second source of truth. The static file is only served when the function has nothing better: no live feed and no backup copy of it. Adding a page to the static file does nothing at all on a normal day.
And when we compared the two lists side by side, the code's list was not just missing the new page. Two other pages, including our contact page, had been missing from it for weeks. Nobody noticed, because nobody reads a sitemap unless something is wrong with it.
The fix that day was a second deploy with the page added to the list in the code. Cheap, once we knew where to look.
Why this reads like sloppiness and is not
When you find two copies of the same list, the first reaction is that somebody was careless. Here, both copies earn their place.
The function is how blog posts reach the sitemap without anyone touching it. The static file is the safety net. If the function fails entirely, the site still serves a sitemap with the main pages in it instead of an error. That safety net has already mattered. During one outage of the blog feed, the sitemap dropped every post, and search engines saw 22 addresses instead of 81 until the function learned to fall back to a saved copy of the feed.
So the design was a decision. The trouble is that the decision was written down in a comment at the top of the function, and the function is not the file you open when you add a page. The person adding a page edits the file called sitemap, which is the reasonable thing to do, and it is the wrong file.
It happened again
We wrote ourselves a note in August: any new page goes in both places, same date. Before publishing this post, we checked whether that note had held.
It had not. The static file lists 24 pages. The list in the code lists 23. One page added on September 11 made it into the file and not into the code, and the live sitemap, 123 addresses long this morning, does not include it. The page itself loads fine and says it is the canonical address, and nothing in the code says leaving it out was on purpose. Google just is not being told about it.
That is the honest lesson. A rule that depends on a person remembering to edit two files will get followed most of the time, and the times it is not followed are silent. We have flagged the missing page to be added. The better fix is to stop having two lists a person can edit: generate the static file from the same list the code uses, or add a check to the deploy that compares them and refuses to ship when they disagree.
What this means for your business
You probably do not have a custom function shadowing your sitemap. You may well have the same shape somewhere else. A website builder that generates the sitemap from pages marked visible, while an SEO plugin keeps its own. A CRM form that writes to one field while a report reads another field with the same label.
The habit that catches all of them is simple. When a live check disagrees with the file you just changed, suspect a second source before you suspect the deploy. Ask what is actually answering that address, not what you edited.
For a Go High Level San Jose business with a blog on the platform and a site somewhere else, open your live sitemap once a quarter. Search it for your newest page and your newest post. If either is missing, the list you are editing is not the list being served.
Want this built for you
We build websites, blogs and CRM systems, and we check them the way search engines see them. Start at optechsol.llc.