Introduction

The robots.txt file is a standard used by web crawlers to determine which areas of your site should or should not be indexed. Proper configuration of this file helps control search engine behavior, protect sensitive areas of your StoreFront, and improve crawl efficiency.

UltraCart automatically serves a default robots.txt file for all StoreFronts, but you can easily override it with a custom version via the File Manager.

Tip: Use the robots.txt file in combination with meta tags (like noindex) and password protection to comprehensively control access to your content.

Example robots.txt

Here’s a basic robots.txt file that blocks specific internal URLs while ensuring that your sitemap is accessible for indexing by search engines.

Download:

Enhanced robots.txt

This version provides explicit allow rules for major LLM’s. If you wish to disallow a specific LLM crawler, change allow to disallow.

You'll notice that UltraCart specifies the URL where the crawler can fetch the sitemap file.  
This helps with making sure crawlers are properly indexing your entire site.

Update the sitemap URL to match the host of your storefront, then upload the robots.txt file to the root folder in the file manager.

Example robots.txt for Development Website

You may want to prevent crawling during development. You can “lock” your storefront (recommended), or use a staging appropriate robots.txt file that disallow crawlers:

Adding the Robots.txt File to Your Storefront

To add the robots.txt file to your storefront:
Navigate to Main Menu → StoreFronts → (Select StoreFront Host) → File Manager.

Open

Navigate: Storefronts > (Storefronts Menu) File Manager

Next, click the new file button:

Open

When the dialog pops up, enter "robots.txt" and click on "OK".

Open

Enter robots.txt file content and click OK.

Open

Best Practices for robots.txt

Here are key recommendations when creating or editing your robots.txt:

• Use User-agent: * to define rules for all crawlers, or specify individual agents (e.g., Googlebot, GPTBot) for more control.

• Block admin or sensitive paths that should not appear in search results.

• Always include a Sitemap directive pointing to your sitemap for improved indexing.

• Avoid blocking CSS or JS resources necessary for rendering the page properly.

Tip: Test your robots.txt with tools like Google Search Console’s Robots.txt Tester.

Considerations for AI and LLM Crawlers

As of now, most large language model (LLM) crawlers do not officially adhere to the robots.txt standard, even if they claim to. However, some do attempt to respect it as a best-effort practice.

Note: These blocks are not guaranteed to be respected but are becoming a growing industry norm. Always review the LLM provider's documentation for up-to-date crawler names.

Conclusion

Using a robots.txt file gives you control over how crawlers interact with your StoreFront. While UltraCart provides a helpful default, you can create a custom file to better align with your SEO strategy or privacy needs. Don’t forget to periodically review and update this file as your site evolves.

Next Steps

• Learn about Search Engine Optimization (SEO) in UltraCart

• Read about Creating Sitemaps

• Google’s robots.txt Specification