Thread Rating:
  • 0 Vote(s) - 0 Average
Share Thread:
What is the limit of a robot.txt file?
#8
Hi,

Yes, the limitations are:

1. The google bot can only read-only first 500KB robot.txt file data.
2. The robot.txt file needs to be added on the top of the domain of your website so that its command gets easily sharable with required details.
3.No assurance that a crawler or robot will visit your site and use your robots.txt file. Actually, search engines use and obey robots.txt file, but malware robots will not.
4. Sometimes the robot.txt file elements get ignored by some search engines


Messages In This Thread

Forum Jump:


Users browsing this thread: 2 Guest(s)
[-]
Ad

[-]
Top Articles
100+ Social Bookmarking Sites List
55+ Image Submission Sites List

[-]
Recent Posts
Starting Consulting Agency
I don't own an agency but I have seen people to struggle with their start-ups. The main...michaelcpleasant — 07:07 AM
HostXNow - All in One Thread
I will take your advice into consideration, I need a new host these days.michaelcpleasant — 06:57 AM
Keyword Clustering
Keyword clustering involves grouping related keywords together based on semantic similari...Shabana — 04:24 AM
Google SERP Changes & Algorithm Updates - April 2024
Many blogs these days don't provide quality content and yet they earn more than small...batman — 02:56 PM
Google SERP Changes & Algorithm Updates - April 2024
@"Kang"#3914 I feel like blogs had it coming. But feel bad though. Many bl...avinash — 02:52 PM

[-]
Follow us on Facebook