|
|
|
Robots.txt Checker
|
|
A User-Agent of search engine is called as Robot. Googlebot is Google's User-Agent.
A robots.txt file provides restrictions to search engine that crawl the web. Robots.txt Checker is a tool which finds the robots.txt file in your site and list out contents of that file.
|
|
|