How Botlight checks.
Every verdict, with a source.
A verdict you cannot verify is a claim, not a measurement. This page lays out everything a report asserts: which crawler tokens are checked and where their names come from, how the four pillars are weighted, why the headline number is capped instead of averaged — and the technical questions the scanner gets asked most often.
The seven answer engines, and the crawlers behind them
One robots.txt line can decide all three columns at once. Training collects pages to train a model — blocking it costs you no visibility. Index builds the retrieval index an assistant cites from — block it and you cannot be cited, no matter how good the page is. Live fetches your page the moment someone asks — block it and the link the assistant wanted to show is dead.
| Engine | Index | Live fetch | Training |
|---|---|---|---|
| ChatGPT OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Google AI Google | Googlebot | — | Google-Extended, GoogleOther |
| Claude Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity Perplexity | PerplexityBot | Perplexity-User | — |
| Copilot Microsoft | bingbot | — | — |
| Meta AI Meta | — | meta-externalfetcher | meta-externalagent |
| Apple Intelligence Apple | Applebot | — | Applebot-Extended |
The names come from the vendors themselves: OpenAI, Google, Anthropic, Perplexity and Apple. A scan additionally reports on fifteen more crawlers — Common Crawl, Amazonbot, Bytespider and the rest — and reads your robots.txt the way the specification prescribes (RFC 9309), not the way it looks.
How the score is built — and why it is capped
Four pillars, weighted: Access 30% (are AI systems allowed to reach you — robots.txt, snippet directives, and whether your CDN serves bots the same page), Legibility 25% (is the content in the HTML, is there enough of it, and is it more substance than repeated boilerplate), Structure 25% (structured data, title and description, canonical, sitemap) and Citability 20% (liftable passages, a named author, a date, sources of your own). Checks that only make sense for certain page types — an author line on an article, product data on a product page — are reported but not scored elsewhere.
The headline number is citability, and it is capped, not averaged: when five of seven engines are not allowed to fetch your page, the cleanest markup does not make up for it — so the total is pushed down to what the closed gate allows, and the report names the condition that took effect. Shops get a second, separate number — the product data score — because what a shopping agent reads from product data is a different question from whether an answer engine may cite a page. A blended number would hide both answers.
A high score means the engines may cite you and can read you — it never promises that they will. The selection happens inside the models, is not measurable from the outside, and any tool that claims otherwise is selling a made-up number.
The technical questions
Does blocking GPTBot take me out of ChatGPT?
No. GPTBot collects pages to train a model. Getting cited runs through OAI-SearchBot, which builds the search index, and ChatGPT-User, which opens your page while the answer is being written. Block those two and you cannot be cited — blocking GPTBot alone costs you no visibility.
Why is Grok not on the list?
Because there is nothing to check. An engine makes this board by bringing one thing: a crawler token the vendor itself publishes, so that a line in your robots.txt verifiably applies to it. xAI documents none — the developer documentation covers the API, not a crawler, and xAI is missing from Cloudflare's reference of major AI crawlers, which lists twenty from twelve operators. The token names in circulation — GrokBot, xAI-Grok, Grok-DeepSearch, xAI-Bot — come from third-party lists that contradict each other, and that contradiction is exactly the point: nobody has a source. A verdict here would be invented, so there is none. The day xAI publishes a token, Grok gets a tile like everyone else.
Does an llms.txt file help?
No vendor has confirmed reading it. It is a proposed standard, and Botlight generates one for you — but it does not count toward your score. Anyone who tells you it is required is guessing.
Back to the scanner — a scan is free, needs no account and runs every check on this page live.