10-08-2018, 06:52 AM
There is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
█ Cheapest VPS | $1 VPS Hosting
█ Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth
█ Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth