02-15-2018, 01:52 PM
Pages: 1 2
02-16-2018, 04:48 AM
Its not utmostly necessary. If your site has a clear navigation then Google and other search engine bots will crawl your pages anyway.
02-21-2018, 11:10 AM
It is not necessary but it is advisable to have a Robot.txt file for your website
03-23-2018, 10:46 AM
Reasons you may want to have a robots.txt file:
You have content you want blocked from search engines
You are using paid links or advertisements that need special instructions for robots
You want to fine tune access to your site from reputable robots
You are developing a site that is live, but you do not want search engines to index it yet
They help you follow some Google guidelines in some certain situations
You need some or all of the above, but do not have full access to your webserver and how it is configured
You have content you want blocked from search engines
You are using paid links or advertisements that need special instructions for robots
You want to fine tune access to your site from reputable robots
You are developing a site that is live, but you do not want search engines to index it yet
They help you follow some Google guidelines in some certain situations
You need some or all of the above, but do not have full access to your webserver and how it is configured
03-26-2018, 08:06 AM
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically Google, bing or yahoo search engine robots) how to crawl and index pages on their website.
03-27-2018, 02:43 AM
I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.
04-02-2018, 06:28 AM
steveposter Wrote:I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.
If navigation is good the search bots will crawl and index anyway.
04-09-2018, 08:55 AM
It is not necessary to use Robots.txt file, But sometime we need to tell crawler which are the pages needs to be indexed and which is not, In that case It is necessary to use Robots.txt file to guide crawler.
04-09-2018, 11:17 AM
Yes robots.txt file is necessary for providing information to search engines. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
09-18-2018, 05:47 AM
It is not necessary to use Robot.txt file in website because it's use only when you require to disallow content or other things.
Pages: 1 2