Thread Rating:
  • 0 Vote(s) - 0 Average
Share Thread:
Is robots.txt file enough to block some pages from Search Engines?
#1
Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.
Reply
#2
No its not enough. Its best that you block them through .htaccess file.
Reply
[-] The following 2 users Like ksmith29's post:
  • danni, lammao
#3
(10-24-2019, 05:52 AM)lammao Wrote: Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.

Robots.text file only tells search engines or suggest some might say that it can or cannot crawl & index any page. It doesn't necessarily blocks them. .htaccess file is the best option for you as @ksmith29 suggested.
Reply
[-] The following 1 user Likes danni's post:
  • lammao
#4
Can anyone provide me the code for htaccess file?
Reply
#5
Follow this article. https://help.dreamhost.com/hc/en-us/arti...ccess-file-
Reply
#6
I hope the article helped you.
SmartScrapers is a data scraping services provider company.
Reply
#7
If it is linked to from other websites, a page that is disallowed in robots.txt can still be indexed. While Google won't slither or record the substance hindered by a robots.txt document, we could in any case find and file a prohibited URL in the event that it is connected from different puts on the web.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)
[-]
Ad

[-]
Top Articles
100+ Social Bookmarking Sites List
55+ Image Submission Sites List

[-]
Recent Posts
Growth Matrix Reviews - Warning! Do NOT Buy Until Knowing This!
Growth Matrix Reviews:>> DISCRIPTION =>> Growth Matrix Reviews is an enhance...RoseBenif — 07:48 AM
Empowering the Skills on Kafka Training with Gologica’s IT Expert
Gologica Offers Kafka Online training, it will help you to get  know  more about Apache K...Naresh1 — 07:35 AM
Nike presenterar den Barcelona-inspirerade Phantom 'Barna'-kollektione...
Nike firar sin koppling till den pulserande staden Barcelona med Phantom 'Barna'-kollekti...fotbollsskorfor — 07:11 AM
PUMA představuje druhé vydání kolekce NJR Creativity Collection, inspi...
PUMA se vrací s druhým vydáním NJR Creativity Collection, pulzující řadou inspirovanou el...kopackycz — 07:08 AM
https://www.facebook.com/slimxmederfahrungen/
SlimXMed Erfahrungen SlimXMed Erfahrungen Weight loss is everyone’s hassle those da...PerezFanders — 07:05 AM

[-]
Follow us on Facebook