How AI systems access your website

AI systems can access websites in several different ways.

Some AI tools can actively browse websites in real time using their own crawlers or search integrations.

For example:

• OpenAI uses the OAI-SearchBot crawler
• Anthropic uses the ClaudeBot crawler
• Perplexity uses the PerplexityBot crawler

These crawlers work similarly to traditional search engine crawlers such as Googlebot or Bingbot.

Some AI tools rely partly on traditional search engines.

For example:

• AI systems may retrieve results from Bing or Google
• AI systems may summarise search engine results rather than directly crawling the website themselves

Some AI models may answer questions using information previously indexed, cached or used during model training.

This means:

• A website may still appear in AI responses even if it is not currently being crawled
• A newly launched website may not yet appear in AI-generated responses

The robots.txt file is a standard file used to provide instructions to website crawlers.

It is available on your ShopWired website at https://yourwebsite.com/robots.txt

The file allows website owners to:

• Block crawlers from certain pages or directories
• Allow crawlers to access certain content
• Specify sitemap locations

Example:

This example tells crawlers not to access the /checkout/ or /account/ areas of the website.

ShopWired does not block major AI crawlers from accessing storefront websites unless:

• You have manually configured your robots.txt file to block them
• Your website is password protected
• Your website or server configuration prevents crawler access

If your website is publicly accessible then AI crawlers can generally access it unless specifically blocked.

Customers sometimes ask AI tools whether they can access a website and receive responses such as:

• "I cannot access this website"
• "I cannot read the robots.txt file"
• "This website is blocked"

This does not necessarily mean the website is actually blocked from AI crawlers.

Possible reasons include:

• The AI model/session does not currently have browsing enabled
• The AI tool failed to retrieve the website temporarily
• The AI misunderstood the prompt
• The AI hallucinated a technical explanation
• The crawler has not indexed the website recently
• The website returned a temporary error

For example, asking:

Can you read my website?

may produce a different response from:

Is my website publicly accessible to AI crawlers?

llms.txt is an emerging proposal intended to help AI systems understand website content.

The proposed file may contain:

• Information about the website
• Important pages
• Documentation links
• Structured summaries

Example location https://yourwebsite.com/llms.txt

However:

• llms.txt is not currently an official internet standard
• Most AI systems do not require it
• Having an llms.txt file does not guarantee visibility in AI responses

At present, llms.txt should be considered experimental rather than essential and ShopWired does not currently create a llms.txt file for your website.

There is currently no guaranteed way to make a website appear in AI-generated answers.

However, the following factors are likely to help.

Your website should:

• Be publicly accessible
• Not require logins for important content
• Not block crawlers unnecessarily

Many AI systems rely partly on search engines and indexed content.

Good SEO practices remain important, including:

• Clear page titles
• Structured headings
• Fast page loading
• Mobile-friendly design
• Internal linking
• Sitemap generation

AI systems are more likely to reference:

• Clear informational content
• Helpful articles
• Detailed product descriptions
• Frequently updated content

AI systems may place greater confidence in:

• Well-known brands
• Frequently referenced websites
• Websites with strong backlinks and authority

ShopWired advises against manually editing your robots.txt file unless you fully understand the impact of the changes. You should also be careful of using AI to generate a robots.txt file for you.

Incorrect rules can accidentally:

• Block search engines
• Block AI crawlers
• Reduce SEO visibility
• Prevent pages from appearing in search results

You can usually check by:

1. Opening the website in a private/incognito browser window
2. Checking whether pages load without logging in
3. Visiting /robots.txt
4. Checking whether important pages are accessible publicly

Some known AI-related crawlers include:

These may change over time.

No.

Allowing crawler access does not guarantee:

• AI visibility
• AI recommendations
• Inclusion in AI-generated answers

AI systems decide independently which sources to use.

ShopWired storefront websites are publicly accessible by default unless restricted through configuration or custom rules.

No, your website does not need an llms.txt file.

It is not required for normal AI crawler access.

Well-behaved crawlers generally respect robots.txt.

However, robots.txt is a voluntary standard rather than a technical enforcement system.

No.

Traditional SEO remains extremely important because many AI systems rely on:

• Search engine indexes
• Structured content
• Website authority signals

AI visibility should currently be considered an extension of SEO rather than a replacement for it.