Last week, during a routine self-audit of our own website, we found something rather awkward: our sitemap had been frozen for 8 days. Not a single article published during those 8 days had made it in.
The awkward part is that nothing reported an error the whole time — posts published successfully, pages opened normally, the active submission API returned success, and the admin panel looked perfectly fine. The only party left in the dark was the search engine, which had no idea we had updated anything.
This is one of the nastiest categories of SEO problem: it throws no errors, affects no traffic, and disrupts no one’s normal use. You will never “discover” it — you can only “check” for it. Today we’ll lay out the common types of silent failures and how to self-audit each one — every item can be checked in a minute or two.
The bottom line first: anything you “configure once and forget” will fail without telling you
In SEO and GEO, most of the work is “one-time configuration” — install a plugin, submit a sitemap, add a snippet of code, enable caching. Once it’s set up and looks fine, nobody ever looks back.
But a website is alive: plugins update, caches expire, publishing methods change, certificates expire, quotas run out. If any single link in that chain quietly breaks, the page still opens as usual — so nobody will ever come and tell you. That is why it has to rely on regular checks, not on “we’ll deal with it when something goes wrong.”
Failure 1: The sitemap stops updating (the one we hit)
Symptom: the newest entry in your sitemap is dated well before the most recent article you actually published.
Why it happens: sitemaps are usually generated by an SEO plugin, and for performance reasons the plugin caches the result. Normally, publishing a new article refreshes it automatically, but some publishing methods don’t trigger that refresh — for example publishing via API, scheduled publishing, or bulk import. Once the cache gets stuck, everything published afterward stays out.
Consequences: search engines mainly discover new content through the sitemap. If the sitemap isn’t updated, new articles have to wait to be found by other means, and indexing is far slower.
How to self-audit (1 minute): open yourdomain/sitemap.xml in a browser and search directly for the URL of your latest article to see whether it’s there. If it isn’t, the sitemap is stuck.
How to fix: clear the SEO plugin’s cache so it regenerates. More importantly, add this step to your publishing workflow — a quick glance after each post is far less trouble than fixing it after the fact.
Failure 2: Structured data was written, but never took effect
Structured data is that “identity description” meant for machines — what the company is called, what this article is, what the FAQs are. It clearly helps how search results are displayed and how AI understands your content, so many companies have added it.
The problem is that adding it doesn’t mean it works. During our own audit we found that several older articles did contain a full block of structured data, but it wasn’t properly wrapped in the tags it belonged in — so it was neither recognized as structured data nor shown on the page. Purely wasted effort, and it had been sitting in the database for months.
Common causes: the editor altered the formatting when the code was pasted in, it was placed in the wrong location, it was stripped out by a minification/optimization plugin, or the web development company added it and never verified it. This kind of problem is 100% detectable — but only if you actually check.
How to self-audit: use the search engine’s official structured data testing tool, paste in the URL, and run it to see whether it can read out the content you expect. If it can’t, the markup isn’t working.
On a related note: if you want AI to state your company name, founding date, and business scope accurately, putting them into structured data is a very high-value step — it’s the form of expression machines are most likely to trust.
Failure 3: The page is blocked by a “don’t crawl me” instruction
This one is the most lethal, and also the most common — especially in the first few months after a new site goes live.
During development, to keep a half-finished site from being picked up by search engines, technical staff usually set it to “do not index.” After launch, they forget to turn it off — so no matter how good the website is, search engines are still being explicitly told “please don’t index me.”
- Check the robots file: open
yourdomain/robots.txt. If you seeDisallow: /, crawling is blocked site-wide. - Check page-level directives: right-click the page and choose “View page source,” then search for
noindex. If you find it, that page has been individually set to be excluded from indexing. - Check the admin setting: many CMS platforms have a checkbox for “discourage search engines from indexing this site” — dig it out and confirm it’s unchecked.
- Check the staging site: if you have a test environment, confirm it is blocked — an indexed staging site will compete with the live site for rankings.
Failure 4: Caching shows you and the outside world two different versions
You edit the content, refresh, and see the new version — no problem. But outside visitors and crawlers may still be seeing the old one.
The reason is simple: you’re logged in, so you usually bypass the cache and always see the latest version, while visitors get the cached version. If the cache wasn’t properly cleared, what you think you’ve fixed hasn’t changed at all on the outside. You updated a price, changed a phone number, took a service offline — and the old version is still out there. This is not rare in the least.
How to self-audit: open the page in an incognito window, or open it on your phone over mobile data (not the company WiFi). If the two don’t match, it’s a caching problem.
One extra reminder: after clearing the cache, warm it up. The first visitor after the cache is emptied has to wait for the server to generate the page on the spot, which can take several seconds — if that person happens to be a customer, the experience is ruined.
Failure 5: A successful submission does not mean successful indexing
Many websites have “active submission” configured — automatically notifying search engines after an article is published. The API returns success, and everyone relaxes.
But a success response only means “the other side received this URL” — not that it will be crawled or indexed. And the submission mechanism itself can quietly fail: the daily submission quota runs out, the key expires, domain verification lapses, or you switch server IPs without adding the new one to the allowlist — and none of these will notify you.
How to self-audit: don’t just look at the submission API’s return value — look at the result. Search for site:yourdomain in a search engine and record the total number of indexed pages; check the webmaster platform to see whether the indexing curve is still rising. If the numbers stop moving for a long time, some link in the chain is broken.
Ten minutes a month: a fixed self-audit checklist
None of these five items requires a technical background — just follow along. We recommend running through them on a fixed day each month, writing down the numbers and watching the trend.
| What to check | How to check it | What normal looks like |
|---|---|---|
| Sitemap | Open yourdomain/sitemap.xml and search for the latest post URL |
The latest post is in there, dated today |
| Structured data | Run the official testing tool on your homepage plus one article page | It reads your company information and article information with no errors |
| Indexing directives | Check yourdomain/robots.txt; search the page source for noindex |
No site-wide Disallow, and no noindex on live pages |
| Cache consistency | Open a page you edited in an incognito window or over mobile data | What you see matches what shows in your dashboard |
| Indexing trends | Note the total from site:yourdomain; check the curve in your webmaster tools |
The total rises slowly as you publish, rather than sitting flat for months |
The key is recording, not checking. A single reading of “320 pages indexed” tells you nothing, because you have no way to know whether that is a lot or a little. But write it down once a month, and after three months the trend speaks for itself: either it is climbing steadily, or it stopped moving in some particular month. A baseline matters more than an absolute number.
A few common questions
Q: Isn’t this the web development company’s job?
A: A web development firm is usually responsible for building the site and delivering it. After delivery the site keeps changing (new content, page edits, plugin updates), and most of these silent failures happen after handoff. Unless you have signed a separate ongoing maintenance agreement, no one comes back to check on a schedule.
Q: What if I find a problem but can’t fix it myself?
A: Find it and write it down first — that step requires no technical skill. Going to a technician with something concrete such as “the sitemap stopped at [date]” or “the structured data testing tool says it can’t read the page” is ten times more efficient than saying “something seems wrong with our website.”
Q: How often should I check?
A: Once a month is enough. If you have just made a major change (new theme, new plugin, server migration, domain change), check the same day you make it — those operations are the most likely to break the items above.
Q: Do these have anything to do with AI search?
A: A great deal. For AI to cite your content, it first has to be able to crawl it and understand it. Being uncrawlable (blocked, sitemap not updated) and being unreadable (structured data not taking effect) are equally fatal on the AI side, and harder to recover from than with a search engine — rank low on a search engine and you are still on page two; on the AI side you simply do not exist.
Worried your own website has problems like these — things that look fine but have actually stopped working?
Siwenlide runs website and SEO/GEO content-acquisition programs for small and mid-sized businesses in Beijing and the surrounding area: we regularly check your sitemap, structured data, indexing directives, caching, and indexing trends, reconnect the links that have broken, and give you a month-by-month comparison record so you can see the changes. We run this self-check on our own site first — the sitemap that had gone 8 days without updating, mentioned at the top of this article, was caught by last week’s check. Visit siwenlide.com (or take a look at our company profile) or call 400-686-2011 to talk it over.
