Joined: May 2017
Posts: 8
Likes Received: 0
Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.
Joined: Jan 2014
Posts: 912
Likes Received: 59
Its not utmostly necessary. If your site has a clear navigation then Google and other search engine bots will crawl your pages anyway.
Joined: Oct 2017
Posts: 57
Likes Received: 0
It is not necessary but it is advisable to have a Robot.txt file for your website
Joined: Nov 2016
Posts: 224
Likes Received: 1
Reasons you may want to have a robots.txt file:
You have content you want blocked from search engines
You are using paid links or advertisements that need special instructions for robots
You want to fine tune access to your site from reputable robots
You are developing a site that is live, but you do not want search engines to index it yet
They help you follow some Google guidelines in some certain situations
You need some or all of the above, but do not have full access to your webserver and how it is configured
Joined: Jan 2018
Posts: 28
Likes Received: 0
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically Google, bing or yahoo search engine robots) how to crawl and index pages on their website.
Joined: Sep 2015
Posts: 673
Likes Received: 5
I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.
Joined: Jan 2014
Posts: 912
Likes Received: 59
steveposter Wrote:I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.
If navigation is good the search bots will crawl and index anyway.
Joined: Feb 2018
Posts: 127
Likes Received: 0
It is not necessary to use Robots.txt file, But sometime we need to tell crawler which are the pages needs to be indexed and which is not, In that case It is necessary to use Robots.txt file to guide crawler.
Joined: Jan 2016
Posts: 725
Likes Received: 1
Yes robots.txt file is necessary for providing information to search engines. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Joined: Jun 2018
Posts: 212
Likes Received: 1
It is not necessary to use Robot.txt file in website because it's use only when you require to disallow content or other things.
Is it necessary to use Robots.txt file?
![[-]](https://seomotionz.com/themes/cerulean/collapse.png "[-]")