---
title: Set crawling behaviour
slug: experience/set-crawling-behaviour
description: Your robots.txt file will tell crawlers how to handle your site, while not all crawlers may interpret it in the same way.
docTags: 
createdAt: 2023-10-18T07:32:19.196Z
---

Your robots.txt file will tell crawlers how to handle your site, while not all crawlers may interpret it in the same way. For detailed information on how google Interprets robots.txt, please refer to this [documentation](https://developers.google.com/search/docs/crawling-indexing/robots/intro) on the matter. 

Locate your robots.txt in your builders' `default/storefront` folder. 

As an example, you could prevent most AI models from accessing your site by adding the following to the robots.txt:

```none
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: AppleBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: YouBot
Disallow: /

```