Thread Rating:
  • 0 Vote(s) - 0 Average
Share Thread:
Is robots.txt file enough to block some pages from Search Engines?
#1
Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.
Reply
#2
No its not enough. Its best that you block them through .htaccess file.
Reply
[-] The following 2 users Like ksmith29's post:
  • danni, lammao
#3
(10-24-2019, 05:52 AM)lammao Wrote: Is robots.txt file enough to block some pages from Search Engines? I mean from crawling and indexing.

Robots.text file only tells search engines or suggest some might say that it can or cannot crawl & index any page. It doesn't necessarily blocks them. .htaccess file is the best option for you as @ksmith29 suggested.
Reply
[-] The following 1 user Likes danni's post:
  • lammao
#4
Can anyone provide me the code for htaccess file?
Reply
#5
Follow this article. https://help.dreamhost.com/hc/en-us/arti...ccess-file-
Reply
#6
I hope the article helped you.
SmartScrapers is a data scraping services provider company.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)
[-]
Sponsored

[-]
Top Articles
100+ Social Bookmarking Sites List
55+ Image Submission Sites List

[-]
Recent Posts
Google SERP Changes & Algorithm Updates - August 2022 (Post 15th)
@"limbo"#1274 Ahhh !!!! You are right. Seems like it.elliekushn — 02:58 PM
Google SERP Changes & Algorithm Updates - August 2022 (Post 15th)
Why there is a certain upward trend for unwanteed kw's from the related ones. is Googl...limbo — 02:56 PM
Live streaming to demonstrate products
You can try it to make your audience believe that the thing is authentic.smartscraper — 10:16 AM
Can a mobile app be used as an affiliate product?
yes sure, why not? It's the best way of marketing of your products.moonlife447 — 09:05 AM
$1 or $2 for each 500 word article, What do you guys say?
As, per my experience.. They are just copy and paste writers. Cheap reviews copies, ...Warner12 — 08:26 AM

[-]
Follow us on Facebook