View and parse the robots.txt file of any website with syntax highlighting.
The robots.txt Viewer on 101IP lets you instantly view and analyse the robots.txt file of any website. The tool shows rules for each User-agent with syntax highlighting, the list of Sitemaps and the Crawl-delay value.
The robots.txt file sits at the root of a site (e.g. https://example.com/robots.txt) and contains instructions for search crawlers: which pages may be crawled and which may not. Following these rules is a voluntary convention.
* means all;