04-09-2018, 08:55 AM
It is not necessary to use Robots.txt file, But sometime we need to tell crawler which are the pages needs to be indexed and which is not, In that case It is necessary to use Robots.txt file to guide crawler.
Users browsing this thread: 1 Guest(s) |
|