04-02-2018, 06:28 AM
steveposter Wrote:I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.
If navigation is good the search bots will crawl and index anyway.