Overview
Purpose
Collects data for Semrush SEO products.
Exact user-agent token
SemrushBot
robots.txt
Allow example
User-agent: SemrushBot Allow: /
Block example
User-agent: SemrushBot Disallow: /
Important limitation
A robots.txt verdict describes the published rule for the tested path. It does not prove crawling, indexing, training, citations or traffic.
Frequently asked question
Does this rule control every product from the same operator?
No. Operators can publish separate user-agent tokens for training, search and user-triggered retrieval. Configure each documented token intentionally.
Official source
Consult the operator’s current documentation before publishing a policy change.