03-25-2019, 07:51 AM
Why the robots.txt file is important. First, let's take a look at why the robots.txt file matters in the first place. The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.