04-09-2018, 08:55 AM
It is not necessary to use Robots.txt file, But sometime we need to tell crawler which are the pages needs to be indexed and which is not, In that case It is necessary to use Robots.txt file to guide crawler.
Is it necessary to use Robots.txt file?
|
« Next Oldest | Next Newest »
|
Messages In This Thread |