A robots.txt file tells search engine crawlers which parts of a website they should or should not crawl. This helps prevent indexing of areas such as admin pages, checkout flows, and other content that should stay out of search results.
In Dynamicweb, robots.txt is generated dynamically from website settings. You do not place a robots.txt file in /Files/System/wwwroot/.
How robots.txt works in Dynamicweb
Each website area contributes its own rules to the generated robots.txt output. In the Advanced tab of the website settings, you can configure:
- Link to sitemap.xml to add a sitemap reference to robots.txt
- Include products in sitemap.xml to include product URLs in the sitemap
- Robots.txt to define crawl rules such as
Disallow: /private/
Example output:
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml
An example of a robots.txt with Link to sitemap.xml enabled and access to /private/ disallowed:
Configure robots.txt
To configure robots.txt:
- Open the Advanced tab of the website settings
- In the Robots.txt section, enter the crawl rules you want to expose
- Enable Link to sitemap.xml if you want to reference sitemap.xml
Dynamicweb serves the generated content on /robots.txt.