What is robots.txt?

[DivyaShravani]

Robots.txt is a text file webmasters create or passing instructions to web robots how to crawl pages on their website.
Example:
User-agent: * Disallow:
User-agent: * Allow: /