+- SEO MotionZ Forum (https://seomotionz.com)
+-- Forum: Search Engine Optimization (https://seomotionz.com/forumdisplay.php?fid=7)
+--- Forum: SEO General (https://seomotionz.com/forumdisplay.php?fid=22)
+--- Thread: What is the limit of a robot.txt file? (/showthread.php?tid=4579)
What is the limit of a robot.txt file? - siva s - 10-04-2018
What is the limit of a robot.txt file?
Re: What is the limit of a robot.txt file? - sinelogixtech - 10-04-2018
Yes, googlebot reads the first 500 KB present in the robots.txt file.
Re: What is the limit of a robot.txt file? - RH-Calvin - 10-08-2018
There is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Re: What is the limit of a robot.txt file? - Ethon Johnson - 10-09-2018
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes (KB)
Re: What is the limit of a robot.txt file? - Mangat Singh Dhiman - 10-29-2018
The limit of robots.txt file in terms of filesize 500 KB .
Re: What is the limit of a robot.txt file? - jackluter - 11-12-2018
there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.
Re: What is the limit of a robot.txt file? - yuva12 - 12-03-2018
Your robots.txt file must be smaller than 500KB.
Re: What is the limit of a robot.txt file? - jonathan brown - 12-14-2018
Hi,
Yes, the limitations are:
1. The google bot can only read-only first 500KB robot.txt file data.
2. The robot.txt file needs to be added on the top of the domain of your website so that its command gets easily sharable with required details.
3.No assurance that a crawler or robot will visit your site and use your robots.txt file. Actually, search engines use and obey robots.txt file, but malware robots will not.
4. Sometimes the robot.txt file elements get ignored by some search engines