AI Crawler Access Audit: Verify Bots with Logs, robots.txt and llms.txt

Your robots.txt allows an AI crawler. A manual request returns 200. The bot may still hit a WAF rule, receive HTML without primary content, or use a different token for the function you care about. The opposite also happens: an official-looking user-agent appears in logs, but its IP does not belong to the provider.

An AI crawler audit is not a copied list of bots. Its job is to reconstruct an evidence chain: what policy you declared, who made the request, what each layer returned, what content the client received, and what happened afterward. Each link answers a different question.

Allowing access does not guarantee a citation. It only removes one possible technical barrier. You must then measure presence, sources, and variance with an AI visibility tracker. Finding no bot in 30 days of logs does not prove a block either: the crawler may not have chosen any of your URLs during that window.

Sources and tokens verified on 2026-08-06. Maintain a dated vendor registry because names, IP ranges, and policies can change.

The Six Layers You Must Not Mix

A useful conclusion separates these states:

An Allow rule covers only the first row. A 200 covers part of the fourth. Neither proves that a page was understood, indexed, retrieved, or cited.

Step 1: Inventory Bots by Purpose, Not Just Provider

Major providers separate functions. A control matrix prevents a training decision from being applied to search or a user-initiated visit.

Check primary sources before changing rules: OpenAI crawlers, Anthropic bots, Google common crawlers, Perplexity crawlers, and Applebot.

Two nuances matter. Google-Extended controls specified Gemini uses without affecting inclusion or ranking in Google Search, and it does not send a separate HTTP user-agent. ChatGPT-User, Claude-User, and Perplexity-User respond to user actions; providers do not all treat robots.txt identically for these fetches. Do not infer a universal rule.

Step 2: Capture the Effective Policy for Every Host

Download https://host/robots.txt from every public host: the root domain, www, documentation subdomains, help centers, and asset hosts if they carry indexable content. Record date, status, Content-Type, redirects, and exact body. One host's policy does not automatically apply to another.

Under the Robots Exclusion Protocol, RFC 9309, the file lives at /robots.txt, the most specific matching rule wins, and an equivalent Allow wins over Disallow. A 4xx may be treated as unavailable and permit crawling; a 5xx can make a compliant crawler treat the site as unreachable. Implementations can vary, so also validate each provider's documentation.

Example of a business policy that allows retrieval while rejecting training where the tokens support that distinction:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

Do not copy this example without deciding your policy and checking current semantics. robots.txt is a public preference, not authorization or security. Protect private content with authentication and real access controls.

Step 3: Test Representative Paths and Conflicts

Evaluating / is not enough. Include:

• a canonical editorial URL;
• a product or pricing page;
• a parameterized URL;
• a deliberately blocked path;
• a client-rendered page;
• a relevant PDF or feed;
• a redirected URL and a true 404.

Calculate the most specific rule per bot. Review wildcards, $ endings, duplicated groups, and generated rules. Include X-Robots-Tag, meta robots, canonical, and authentication.

Step 4: Export Edge and Origin Logs

An initial window of 30 to 60 days is usually practical. Gather CDN, WAF, load balancer, and origin events where available: a request blocked at the edge never reaches the application server.

Minimum fields:

Normalize variants and IP rotation while preserving raw events. Group by provider, purpose, host, template, status, and day. Never combine unverified and official traffic.

Step 5: Verify Identity Before Allowing Exceptions

A user-agent is text controlled by the requester. Do not use it as the only criterion for opening your WAF.

Follow each provider's official mechanism:

1. compare the source IP with current published ranges;
2. when instructed, run reverse DNS and confirm it with a forward lookup;
3. use verified bot signatures if your CDN supports them;
4. also require the user-agent to match the declared purpose;
5. store method, registry version, and result with the event.

Google documents verification with ranges or confirmed DNS. OpenAI and Perplexity publish separate lists for their tokens. If you create an exception, combine verified identity, purpose, method, paths, and reasonable limits. Never allow globally because a name contains bot or a familiar brand.

Step 6: Classify Access with Operational States

A binary allowed/blocked table hides problems. Use states that lead to action:

Measure requests and unique URLs, but also success rate, status distribution, median bytes, and affected templates. One hundred 200 responses of 900 bytes can be one hundred empty shells, not one hundred useful pages.

Step 7: Inspect the Payload, Not Only the 200

Capture the response delivered to a client without a session and compare it with the browser. Look for:

• title, H1, and primary content in initial HTML;
• consistent canonical and hreflang;
• relevant Article, BreadcrumbList, and other structured data;
• crawlable internal links with href;
• no challenge, consent wall, or error message;
• key resources without authentication or expiring URLs;
• true status for 404s and redirects, not a generic SPA 200.

Correct schema helps express entities but does not repair a block. Audit that layer separately with the schema markup and structured data guide for AI.

Step 8: Run Controlled Tests and Join the Layers

The goal is not to emulate all provider behavior, but to isolate your infrastructure. For a test URL:

1. request it without cookies from outside your network;
2. use a control user-agent and a candidate one without claiming official identity;
3. record timestamp and request ID;
4. confirm the event in CDN/WAF and origin;
5. capture status, headers, bytes, and body;
6. compare it with the expected robots.txt rule;
7. repeat after remediation and retain both pieces of evidence.

Do not spoof an IP or present this as a real provider visit. It helps reveal rules conditioned on user-agent, geography, cache, or session. Definitive evidence of an official bot still requires an observed and verified request.

Where llms.txt Fits and Where It Does Not

The llms.txt proposal defines a Markdown file that can guide language systems to useful documentation and pages. In this audit, you can check:

• a 200 response and reasonable Content-Type;
• a clear title and summary;
• absolute or resolvable links;
• no private, broken, or non-canonical URLs;
• consistency with navigation, sitemaps, and current content.

Do not use it to grant permission, block training, or demonstrate adoption. The proposal itself does not define how every system must process it. llms.txt does not replace robots.txt, sitemaps, canonical, security, or accessible HTML; it is not evidence of ranking or citation either.

Remediate with Acceptance Criteria

Every fix should declare initial state, change, scope, and closing test. Example:

Change WAF rule `managed-bot-14` to allow `OAI-SearchBot` only when IP and user-agent pass official verification, on public GET/HEAD routes, while retaining limits and authentication blocks. Accept when a controlled test returns complete HTML and a later official request appears as verified and 200.

Prioritize in this order:

1. errors that block the whole host or serve false responses;
2. policies that contradict the business decision;
3. accidental WAF, CDN, rate limiting, and authentication blocks;
4. incomplete HTML, redirects, and incorrect statuses;
5. canonical, hreflang, link, and schema consistency;
6. optional discovery files.

Move each finding into the AI visibility action backlog with an owner, evidence, and review date. Do not close it at configuration time: observe logs for a sufficient window and preserve a comparable sample.

Audit Checklist

• [ ] Dated inventory of providers, tokens, purposes, and official sources.
• [ ] robots.txt captured per host with status, headers, and body.
• [ ] Representative paths evaluated with the most specific rule.
• [ ] CDN, WAF, and origin logs joined by request ID.
• [ ] Verified and unverified identities reported separately.
• [ ] Status, bytes, latency, cache, and edge actions classified.
• [ ] Initial HTML and key resources inspected without a session.
• [ ] 404s, redirects, canonical, hreflang, and schema checked.
• [ ] llms.txt treated as an optional map, not permission.
• [ ] Fixes have acceptance criteria and scheduled re-observation.
• [ ] Later mention and citation measurement is separate from technical access.

FAQ

How do I know whether an AI crawler actually accessed my site?

Find the request in CDN, WAF, or origin logs and verify its identity with the provider's official method. Then inspect host, path, status, bytes, latency, and the content served. A user-agent can be spoofed, while no requests in the window does not prove a block if the bot never attempted to crawl.

Is allowing the user-agent in robots.txt enough?

No. robots.txt declares a policy for crawlers that honor it, but an allowed request can still be blocked by a CDN, WAF, authentication, rate limiting, or application errors. It may also receive empty or incomplete HTML. The audit must compare policy, observed access, and payload.

What is the difference between search, training, and user-initiated bots?

They serve different purposes and commonly use separate tokens. OpenAI distinguishes OAI-SearchBot, GPTBot, and ChatGPT-User; Anthropic distinguishes Claude-SearchBot, ClaudeBot, and Claude-User. A rule for a training bot should not automatically be interpreted as a rule for search or a visit initiated by a person.

Should I allow an AI bot just because its user-agent looks official?

No. Combine the user-agent with the official mechanism available: published IP ranges, forward-confirmed reverse DNS, or bot signatures. Store the verification result in the log. Allowing by user-agent name alone lets an unauthorized actor impersonate the crawler.

What is llms.txt used for in an audit?

It can act as an optional Markdown map to content useful to language systems. The audit checks that it responds, its links work, and it matches the information architecture. It is not access control, does not replace robots.txt or a sitemap, and alone does not prove provider adoption or improve rankings.

Does allowing AI crawlers guarantee that my site will be cited?

No. Access removes one possible technical barrier, but source selection also depends on relevance, quality, corroboration, freshness, and the specific system. After fixing a block, preserve the date and repeat a comparable measurement of mentions and citations to observe whether the outcome changes.

Turn Technical Access into Evidence, Not Assumptions

The useful question is not "do we allow GPTBot?" but "which function do we want to enable, which identity did we verify, what response did it receive, and what did we observe afterward?". A well-closed audit leaves a deliberate policy, classifiable logs, secure exceptions, and repeatable tests.

Once the technical barrier is resolved, measure separately whether mentions, sources, and citations change. That distinction tells you whether you fixed a necessary condition or achieved an outcome that still depends on relevance and selection.

Measure your brand's AI visibility with Mentio ->

Want to know if AI mentions your brand?

Discover your visibility in ChatGPT, Claude and Gemini in minutes.

Related articles