What's the significance of Robots.txt file in a website?

[DivyaShravani]

Robots.txt generator tells the search engines which pages to access and index on your website and which pages not to index on your website. In Robots.txt we give user agent * Here User indicates the US, and the Search engine is our agent by (* Mark ) we are saying them to allow or Disallow the coding language.

For Example :

User-agent: *
Disallow: /wp-admin/
Allow: /wp-content/uploads/