Joined: Nov 2013
Posts: 557
Likes Received: 0
Which is better Robots.txt or Meta Robots Tag?
Joined: Jan 2016
Posts: 725
Likes Received: 1
Robots.txt is better. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Joined: Mar 2017
Posts: 175
Likes Received: 0
Robots.txt dictates site or directory-wide crawl behavior, whereas meta robots can dictate indexation behavior at the individual page level.