03-29-2018, 07:06 AM
Robots.txt is a text file webmasters create or passing instructions to web robots how to crawl pages on their website.
Example:
User-agent: * Disallow:
User-agent: * Allow: /
Example:
User-agent: * Disallow:
User-agent: * Allow: /