Set crawling behaviour
1 min
Your robots.txt file will tell crawlers how to handle your site, while not all crawlers may interpret it in the same way. For detailed information on how google Interprets robots.txt, please refer to this documentation on the matter.
Locate your robots.txt in your builders' default/storefront folder.
As an example, you could prevent most AI models from accessing your site by adding the following to the robots.txt:
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: AppleBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: YouBot
Disallow: /