Why does optimizing for AI answers still lose you readers?
You updated your site for AI crawlers and answer engines, yet models still describe your brand wrong. The machine-facing web you were told to build has no settled rules, and the rules that exist are ignored by most of the bots that matter.
Google says the obvious fix is speculative
The clearest signal is that Google itself sends mixed messages. Search Engine Journal reports that a Reddit discussion flagged seemingly conflicting guidance about LLMs.txt, an AI-oriented file format, between Google properties. Mueller's answer, per SEJ: LLMs.txt is purely speculative for now. He points instead to WebMCP, a Google-backed proposal, as what site owners should watch. When the biggest gatekeeper has no settled standard, betting your roadmap on any single AI file is premature.
- Speculative format plus conflicting guidance means no safe file-based bet
- WebMCP, a Google-backed proposal, is Mueller's suggested direction to watch
The bots you can't negotiate with
Even if you follow every declared protocol, most AI access happens outside it. Digiday describes a visible layer of declared good bots that pass through infrastructure gatekeepers like Cloudflare, Fastly and Akamai. But a much larger gray crawling economy sits largely outside meaningful oversight, and Digiday reports the black market scraping economy still wins. Cloudflare's new rules for mixed crawlers, bots used for both search and AI training, show gatekeepers trying to catch up. Your robots.txt governs the polite minority, not the majority that actually shapes how models see you.
- Good bots identify themselves; gray scrapers do not
- Mixed crawlers blur the line between search indexing and AI training
- The oversight gap is structural, not a settings problem you can fix
Being read is not being understood
Machine readability gets you cited, not correctly represented. Harvard Business Review's research on luxury brands finds models reliably process explicit cues but face a distinct visibility and interpretation risk. HBR's advice: structure product data for machine readability, seed authoritative language in owned and earned media, and monitor your share of model across platforms. That means writing clearly for humans first, because authoritative human-facing language is what the models are reading.
- Explicit facts travel well into AI answers; brand nuance does not
- Explicit facts travel well; implicit brand meaning does not
What this does not solve
None of this gives you a guarantee, and that is the point. Per SEJ, Google may never standardize on LLMs.txt or WebMCP. Per Digiday, gray scrapers will keep taking content regardless of your settings. Per HBR, monitoring share of model shows how you appear in answers today, not how to control them. Publishing for both humans and machines reduces risk; it does not eliminate it.
- Measurement and control are different problems
- Infrastructure rules cover only declared bots
Do you know your share of model: how do AI answers currently describe your brand?
If gray scrapers ignore robots.txt, which content would you still publish openly?
Ask one AI assistant about your own brand today and log where its answer is wrong.
Sources
Recommended for you
- GEO
Why does your brand vanish from AI answers?
AI engines now answer buyers directly, skipping your site entirely. Rankings no longer protect you from disappearing.
- GEO
Why can't you measure where AI referrals actually come from?
AI assistants now send real traffic and ad-driven conversions, but most funnels still can't connect an AI mention to a paying customer.
- GEO
Why does AI search traffic still look worthless in your analytics?
AI-referred visitors may convert far better than search visitors, but your dashboard labels them as direct or unknown.
- GEO
Why does GEO still leave your brand invisible in AI answers?
Generative engine optimization promises citations inside ChatGPT and AI Overviews. The evidence on whether it reliably works is thinner than the sales pitch suggests.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.