---
title: robots.txt
slug: experience/robotstxt
docTags: 
createdAt: 2026-09-30T10:29:36.840Z
---

`robots.txt` tells search engines and other crawlers which parts of your site they may crawl. Put it at:

:::BlockQuote
web/default/storefront/public/robots.txt
:::

Example: allow everything except the account pages, and point crawlers to your sitemap:

:::BlockQuote
User-agent: \*
Disallow: /account/

Sitemap: https\://www\.example.com/sitemap.xml
:::

Example: block everything, e.g. while a site is still being prepared:

:::BlockQuote
User-agent: \*
Disallow: /
:::

### Good to know

- **No file means "allow all".** Without a `robots.txt`, your custom domain answers with `404`, which crawlers treat as "everything allowed".
- **Moving from the old location.** Before PXP 6.1, `robots.txt` was located at `default/storefront/robots.txt`. The upgrade to 6.1 moves it into the public folder (`web/default/storefront/public/`). Until PXP 7.0 the old location is still served if the file wasn't moved.