robots.txt
2 min
robots.txt tells search engines and other crawlers which parts of your site they may crawl. Put it at:
web/default/storefront/public/robots.txt
Example: allow everything except the account pages, and point crawlers to your sitemap:
User-agent: *
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
Example: block everything, e.g. while a site is still being prepared:
User-agent: *
Disallow: /
Good to know
- No file means "allow all". Without a robots.txt, your custom domain answers with 404, which crawlers treat as "everything allowed".
- Moving from the old location. Before PXP 6.1, robots.txt was located at default/storefront/robots.txt. The upgrade to 6.1 moves it into the public folder (web/default/storefront/public/). Until PXP 7.0 the old location is still served if the file wasn't moved.