If AI can’t read your site, AI can’t cite your site. And strategies that used to work for SEO might not be working anymore for AI crawlers.
Why this is suddenly a problem
There’s two main issues.
First, making sure that you’ve allowed AI to find your site. With the proliferation of spam and malicious bots, security best practices try to keep the bad stuff out. But they might be keeping the new good stuff out too.
And second, many modern websites load their content using code that runs in your browser (often called JavaScript). That can create beautiful, fast experiences for people.
But the source notes a tough detail: among the crawlers used by these AI systems, only Google’s AI (Gemini) currently supports reading pages that rely on that browser-run code.
So your site might look perfect to humans and still look empty to AI crawlers.
What to do if your site loads content “after the page appears”
If your content shows up only after the page runs browser code, you may need to create a version that bots can read right away. The source calls out approaches teams used years ago for search engines, and they’re useful again now:
- Build pages so the readable content is available immediately (not only after scripts run)
- Generate a ready-to-read version ahead of time (this is called server-side rendering)
- Provide a simple fallback version of key content so bots can still see it
If you use a common website platform, you’re probably fine, but confirm
Iff your site is built on a standard website platform like WordPress or Drupal, you’re likely okay, because content is typically readable to crawlers. Still, it’s worth checking. And if you hear the term “headless”, all bets are off, because it completely depends on the developer.
Check robots.txt (it might be blocking AI by accident)
Most sites have a file called robots.txt. It tells bots what they can and can’t crawl.
But bots used by AI tools are not the same as the ones used by traditional search engines. Rules you set for performance or security might accidentally block AI crawlers. Which doesn’t mean all bots from AI companies, because some of them could be training bots, and you might not want them trained on your content.
Examples of “citation bots” you want to make sure are getting through include OpenAI’s ChatGPT, Anthropic’s Claude-Web and Claude-SearchBot, and Perplexity’s PerplexityBot and Perplexity-User.
Add llms.txt (a simple “here’s what matters” guide for AI)
This isn’t universal, but some AI bots (and for sure Anthropic, the company behind Claude) read a proposed standard known as llms.txt. (Google says they ignore it, but they index it anyway).
Plain-English version: llms.txt is a short, AI-friendly map of your site. It helps an AI crawler understand:
- What your site is for
- What pages matter most
- Where the best content lives
There are simple formatting conventions (headings, bullet lists, links, and short descriptions next to links). It’s not guaranteed that it will help with all AI platforms, but it won’t hurt, and could have increasing upside. Include at least your main sections and the next level down. Minimal effort, potentially big upside.
Where to start
- Confirm whether your key pages show readable content without relying on browser-run code
- Review robots.txt to make sure you’re not blocking AI crawlers by accident
- Publish an llms.txt file listing your most important pages with short descriptions
Bottom line
Once AI can reliably read your site, your content can start working harder for you to drive AI-generated traffic.
