ScoutBot

A small research crawler for e-commerce market analysis.

What it reads

• Public storefront data on Shopify stores that are running public ads: the homepage, /meta.json and /products.json.
• Nothing behind a login, no checkout, cart or customer data, and no form submissions.

How it behaves

• One store at a time, at least two seconds between requests, and an adaptive slowdown on any HTTP 429.
• It honours robots.txt (including Crawl-delay), using the ScoutBot group if present, otherwise *.
• It always honours Retry-After.
• Requests are signed with Web Bot Auth. Key directory: /.well-known/http-message-signatures-directory
• User-Agent: ScoutBot/1.0 (+https://crawler.muhammadmuneeb.tech)

Opt out or contact

To have your store excluded, or to report a problem, email crawler@muhammadmuneeb.tech. Exclusions are applied within 24 hours.