How to manage AI crawler access without losing discoverability

AI crawler access is no longer a simple yes or no decision.

For many businesses, the website needs to remain easy for search engines and AI assistants to understand. Service pages, articles, FAQs and business information should be discoverable when a potential customer asks a relevant question.

That does not automatically mean every crawler should be allowed to collect every page for model training.

This is where the decision has become more useful, and more delicate.

The useful distinction

Cloudflare now separates AI bot policies into three broad behaviours: Search, Agent and Training. Search crawlers collect or index content so it can be found later. Agent traffic acts in real time on behalf of a person. Training crawlers collect content to train or fine tune a model, including mixed purpose crawlers that combine search and training behaviour.

That distinction matters because most business websites do not have one single preference.

An informational website may want its service pages available to search systems and AI answer engines. It may want agents to fetch public pages when a real person is asking about the business. It may still prefer not to allow general model training on its content.

Treating all AI crawlers as one category makes that decision blunt.

What changed

On August 21, 2026, Cloudflare announced Bot Preference Sync, a feature that keeps Cloudflare AI bot preferences reflected in robots.txt.

Instead of configuring one policy at the edge and maintaining a separate static robots.txt file, site owners can keep the published preference aligned with the policy they have already set for AI bot categories. Cloudflare says the feature is available across Free, Pro, Business and Enterprise plans.

For a small business, that is useful because the risk is often not one dramatic crawler mistake.

The risk is drift.

Someone blocks a crawler in one place. A stale robots.txt file says something else. A sitemap still exposes pages that the business no longer wants crawled. A future website change overwrites the file. Nobody notices until traffic, referrals or content access behaves differently from what the business expected.

Bot Preference Sync does not remove the need for judgement, but it can reduce one avoidable source of inconsistency.

Robots.txt is still only a preference

The most important limitation has not changed.

Cloudflare's own robots.txt documentation is clear that robots.txt expresses crawler preferences. It does not technically prevent access. Cooperative crawlers may respect it. Others may ignore it.

That means robots.txt is useful for signalling intent, but it should not be mistaken for enforcement.

If the business genuinely needs to prevent certain crawler access, the control needs to happen closer to the edge, using Cloudflare AI Crawl Control, WAF rules or another suitable enforcement layer. The published preference and the enforced rule should agree.

The practical model is simple:

• Use robots.txt to state the policy.
• Use edge controls to enforce the policy where blocking matters.
• Use analytics and logs to verify the policy is behaving as intended.

The mistake is relying on only one of those layers and assuming the job is done.

A sensible starting policy

For many Australian SMB websites, the starting point will look something like this.

Allow important public content to remain available for Search. That includes service pages, location information, helpful articles, case studies where appropriate and contact pathways. If the website is part of how the business earns trust, it should be understandable by the systems people use to search.

Allow Agent access where it supports real user tasks. That might include letting an assistant fetch a public service page, summarise a visible FAQ or help someone compare options. It does not mean every automated request should be trusted.

Disallow Training where the business has no clear reason to grant it. Training access is a separate decision from being discoverable. A business can want visibility without granting broad reuse of its content for model development.

Then test the result.

That last step is easy to skip. It is also where the real confidence comes from.

What should be audited

A practical AI crawler and discoverability audit should inspect more than one setting.

Start with the visible crawl signals. Review robots.txt, sitemap output, canonical URLs, redirects, noindex rules, HTTP status codes and whether important pages are reachable without scripts that hide the core content.

Then review Cloudflare settings. Check Search, Agent and Training preferences. Confirm whether Bot Preference Sync is enabled. Look for older bot settings that may overlap with newer AI bot policies. Cloudflare notes that it is updating defaults for new domains from September 15, 2026, with Search remaining allowed while some Agent and Training behaviour is blocked on pages with ads.

Next, inspect the website content itself. AI search visibility depends on clear HTML content, structured headings, useful internal links, accurate business information, schema where appropriate and pages that explain the business in plain language. Blocking training crawlers does not compensate for thin or confusing content.

Finally, check measurement. Look at referral traffic, AI crawler requests where available, important landing pages and whether analytics can separate useful discovery from automated noise.

The result should be a written policy, not just a toggle.

The risk of overblocking

Some businesses will hear "AI crawler" and block everything.

That may be reasonable for private, licensed or publisher content. It is not automatically the right move for a service business that wants to be found.

Overblocking Search or useful Agent traffic can make the website harder for AI answer systems to understand. That can reduce the chance that the business appears when people ask questions the website is well placed to answer.

The opposite risk is also real. Allowing everything can expose content to uses the business did not intend, especially if nobody has reviewed which pages should be public, which should be indexed and which should be protected.

The better path is intentional access.

The website still has to be understandable

Crawler controls are only one part of AI Search Optimisation.

If the site does not clearly explain who the business serves, what it provides, where it operates and how clients should engage, crawler settings will not fix that. AI systems can only work with the content and structure they can access.

For Andy Vu Lab, that means the technical policy should support the content strategy.

Important service pages should remain crawlable. Useful articles should be easy to understand and cite. The sitemap should be accurate. llms.txt should stay concise and factual. Structured data should match visible content. Referral measurement should help identify whether AI assisted discovery is creating useful enquiries or just noisy traffic.

This is usually Website Growth & Care work as much as search work. It belongs in the ongoing operating rhythm of the website.

A practical audit checklist

Before changing crawler policy, ask:

• Which pages must remain discoverable through search and AI answers?
• Which pages should not be available for model training?
• Are Search, Agent and Training preferences configured separately?
• Does robots.txt match the edge level policy?
• Are old bot rules conflicting with newer crawler settings?
• Do sitemap, canonical and noindex rules still match the content strategy?
• Can important service pages be fetched and understood without fragile client side behaviour?
• Are AI referrals and crawler activity visible enough to review later?
• Who owns this policy after the initial setup?

The final question is the one that keeps the policy healthy.

Crawler behaviour will keep changing. Platform defaults will keep changing. Search behaviour will keep changing. A useful website needs a maintained decision, not a one time preference file.

The next decision

AI crawler management sits between visibility, content protection and technical governance.

That makes it a good candidate for a focused Website Audit, especially for businesses already using Cloudflare or planning more serious AI search work.

The useful outcome is not a perfect crawler policy. That would be false precision.

The useful outcome is knowing what you are allowing, what you are disallowing, what is actually enforced and whether important pages remain discoverable.

That is the level of clarity most businesses need before AI search becomes another source of confusion.