AI Crawler Rules: Keep Search and Training Choices Separate

Can you keep your business findable without allowing every AI use of its pages? Start by separating search access from training controls. The crawler's name and purpose matter more than a broad label such as AI bot.

Decide what you want customers to find

Your opening hours and service area help a customer decide whether to contact you. A paid report or private customer record has a different purpose. Choose what belongs on public pages before deciding who should crawl them.

I wouldn't block a useful public service page just because its information might appear in an AI search answer. I also wouldn't tell a business to publish material it needs to keep private.

That's a judgment about the page's job. It doesn't settle which uses of its content you want to allow.

Check the actual crawler

Google's AI search documentation separates controls that are easy to lump together. Googlebot provides access for Search, including AI Overviews and AI Mode. Google-Extended controls some other AI training and grounding uses.

Google-Extended doesn't control inclusion or ranking in Google Search. Blocking it is therefore a different choice from blocking Googlebot. Check the current guidance and your settings before applying a rule to both.

Other services have their own crawler names and controls. Don't assume the Google distinction describes all of them. Identify the service, its purpose and the precise control your host or firewall applies.

Letting a crawler in is only one part

Access permits a request; it doesn't promise indexing, ranking or a mention in an AI answer. Your service page still needs to explain what you do and where you do it.

Say whether you serve a town, offer a particular repair or answer calls outside office hours. Use what your business actually provides. Leave out claims that depend on a schedule you haven't confirmed.

The sample rewrite in the writing guide shows how a vague paragraph can become a usable answer. It illustrates clearer wording. It doesn't demonstrate a particular search outcome.

Check whether the visit helps someone act

A useful visit can end with a call, directions or a completed request. An AI mention that sends nobody to the right place hasn't answered whether your website is helping.

The traffic and leads section explains what to record. Compare real enquiries with visits over time. Don't credit a crawler change for an increase you can't trace.

If your current page already answers the customer's question, keep it. The needed work may be one crawler setting or a clearer contact route. It doesn't automatically call for a new website.