+- SEO MotionZ Forum (https://seomotionz.com)
+-- Forum: Search Engine Optimization (https://seomotionz.com/forumdisplay.php?fid=7)
+--- Forum: SEO General (https://seomotionz.com/forumdisplay.php?fid=22)
+--- Thread: What is robots.txt? (/showthread.php?tid=3646)
Pages:
1
2
What is robots.txt? - WebExpert - 02-28-2018
hello guys, if anyone has knowledge about the robots.txt, Please share, because I don't know anything about it.
Re: What is robots.txt? - komali - 02-28-2018
Robots.txt file is used to give instructions to bots index or not. for this, you need to submit in that file neither index or not for particular search engine bots.
Ex: User-agent: * Disallow:
User-agent: * Allow: /
Re: What is robots.txt? - limbo - 03-01-2018
If you have a proper sitemap then its irrelevant, search engine bots will crawl your site anyway unless you want to block some of your urls.
Re: What is robots.txt? - jackluter - 03-02-2018
Robot.txt is a file which helps to secure your website payment page and security information on your website then you should submit to CMS after that crawlers and robots they did not crewel your secure information
Re: What is robots.txt? - rachana - 03-08-2018
Robots.txt is a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.
Re: What is robots.txt? - andreausa - 03-12-2018
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Re: What is robots.txt? - DivyaShravani - 03-29-2018
Robots.txt is a text file webmasters create or passing instructions to web robots how to crawl pages on their website.
Example:
User-agent: * Disallow:
User-agent: * Allow: /
Re: What is robots.txt? - blackmagicbaba - 03-29-2018
Agree with you guys.
Re: What is robots.txt? - anneroberts - 04-03-2018
Robot.txt file contains standard way to inform the web robot about which areas of the website should not be processed or scanned.
Re: What is robots.txt? - RH-Calvin - 04-09-2018
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.