Crawler policy
A documented decision on GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and others, implemented in robots.txt.
You getCrawler policy and robots.txt
LLM SEO · USA
LLM SEO for what models know and what they look up
Large language models answer from two places: what they learned in training and what they retrieve at question time. Good LLM SEO works on both.
- 6 AI platforms
- 60 buyer prompts
- No obligation
Where a model’s answer comes from
Learned in training
Brand mentions, reviews and articles from before the cut-off
Fetched live
Pages its search crawler can reach today
Your Brand named, with today’s facts
In short
LLM SEO is the work of shaping how large language models understand and present your business. Models answer from training data, which reflects the web at some past point, and from live retrieval, which fetches pages at question time. Retrieval responds to crawler access and clear pages; trained knowledge responds to consistent entity facts and mentions across many credible sources.
- Two sources
- Training data and live retrieval
- Retrieval crawlers
- OAI-SearchBot, PerplexityBot
- Training crawler example
- GPTBot
- Key lever
- Consistent entity facts
Why it matters
Why LLM SEO has two sides
ChatGPTAI OverviewsAI ModeGeminiPerplexityCopilot
Ask a model about your company and the answer may come from memory, from a fresh search, or both. Memory comes from training data and can be out of date. Retrieval reads current pages, but only the ones its crawler can reach and its search system selects. A business can be missing on one side and well represented on the other.
That's why the same question can produce a confident but outdated description on one platform and an accurate, cited one on another. LLM SEO looks at both halves: making sure retrieval can find your current pages, and making sure the wider web describes you consistently enough that future training reflects reality. Neither half alone is enough.
How it works
How LLM SEO works
Four levers, split between what models remember and what they fetch.
- 01
Crawler access decisions
Decide which bots to allow. Search and retrieval crawlers affect current answers; training crawlers affect what future models may learn.
- 02
Entity consistency
One accurate description of your business, repeated across your site, profiles and directories, gives models a stable picture to learn from.
- 03
Mentions on trusted sources
Industry publications, associations, reviews and partner pages are how a model learns that you exist and what you're known for.
- 04
Retrievable answers
Clear pages on your own site give retrieval systems something accurate to fetch and cite at the moment of the question.
What we do
Our LLM SEO service
Work aimed at both what models already know and what they can look up.
Model knowledge check
We ask each platform what it knows about your business and separate memory errors from retrieval errors.
You getAccuracy findings by platform
Entity fact sheet
Your canonical business facts, applied across your site, schema and key profiles.
You getApproved fact sheet
Mention building
Targeted outreach to the publications, directories and associations relevant to your industry.
You getMention plan and progress log
Retrieval-ready pages
Pages that answer your buyers' questions directly, so live answers have accurate material to work with.
You getRevised priority pages
Measurement & reporting
Your 60 prompts tracked across six platforms, with notes on what changed and why we think it did.
You getMonthly AEO scorecard
Side by side
Training data vs live retrieval
| Training data | Live retrieval | |
|---|---|---|
| When it's gathered | Before the model is released | At the moment of the question |
| How current | Can be months or years old | As current as the page fetched |
| Example crawler | GPTBot (OpenAI) | OAI-SearchBot (OpenAI), PerplexityBot |
| Shows citations | No | Usually, with links |
| What influences it | Consistent facts and mentions across the web | Crawler access, indexing and clear pages |
| When changes show | Only in future model versions | Once pages are fetched again |
Most AI search answers mix both, which is why we work on both.
Working with us
Starting an LLM SEO project
No lock-in surprises. You see the plan, the work and the results.
Get an AEO quote- Phase 1
Free check
A fast look at how AI platforms describe your business today, before any commitment.
- Phase 2
Scope call
We agree on prompts, platforms and where you stand on training crawlers, then quote.
- Phase 3
Audit
Model knowledge, retrieval access and entity consistency reviewed side by side.
- Phase 4
Execution
Crawler policy, fact cleanup, mentions and pages, with re-tests every month.
What is LLM SEO?
It's optimizing for how large language models understand and describe your business. It covers both the knowledge models pick up in training and the pages they retrieve when answering a question.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's crawler for training data. OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI says sites that block it won't appear in those search answers. They're separate settings, so you can allow one and block the other.
Should I let AI companies train on my website?
It's a business decision. Some companies block training crawlers to protect original work; others allow them so models learn accurate information about them. Either way, keep search crawlers allowed if you want to be cited.
Can I change what ChatGPT already knows about my company?
Not directly. You can make sure live retrieval finds accurate pages, and correct the sources spreading wrong information, so answers improve as models and indexes refresh.
Is LLM SEO the same as AEO or GEO?
They overlap heavily. LLM SEO emphasizes the model side, meaning training and entity understanding, while AEO and GEO emphasize answers and citations. The underlying work is largely shared.
How do mentions help with LLM SEO?
Models learn about businesses from how consistently they're described across credible sources. A feature in a respected Atlanta trade publication or a listing in a state association directory tells a model more than one more page on your own site.
Find out what AI models believe about you
Start with the free check and we'll separate outdated memory from retrieval gaps you can fix.