06-08-2017, 07:09 AM
You can try checking these, I think they have decent explaination how it works:
<!-- m --><a class="postlink" href="http://www.robotstxt.org/robotstxt.html">http://www.robotstxt.org/robotstxt.html</a><!-- m -->
<!-- m --><a class="postlink" href="https://moz.com/learn/seo/robotstxt">https://moz.com/learn/seo/robotstxt</a><!-- m -->
<!-- m --><a class="postlink" href="http://www.robotstxt.org/robotstxt.html">http://www.robotstxt.org/robotstxt.html</a><!-- m -->
<!-- m --><a class="postlink" href="https://moz.com/learn/seo/robotstxt">https://moz.com/learn/seo/robotstxt</a><!-- m -->