• 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why the robots.txt file is important ??
#1
Answerpail.com Question and Answer Website
Hello Friends,

Why the robots.txt file is important ??
Reply
#2
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. The slash after “Disallow” tells the robot to not visit any pages on the site.
Reply
#3
It is very important for indexing a website. We can control robots (search engine crawler) through it. Suppose, I want to block some page to the indexing of a website. Then I can do it through it.
Reply
#4
Robots.txt file let search engines know which page to crawl & which page not to crawl. You can control the indexing of your website through Robots.txt file.
Reply
#5
Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not index.
Reply
#6
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Windows VPS | Windows with Remote Desktop Access
Cheap Dedicated Server | $29 Dedicated Server with Free IPMI
Reply
#7
(06-27-2018, 10:04 PM)pharmasecure Wrote: It is very important for indexing a website. We can control robots (search engine crawler) through it. Suppose, I want to block some page to the indexing of a website. Then I can do it through it.

Exactly,We can block some URL with the help of robots.txt file....
Reply
#8
(06-27-2018, 04:02 AM)LitchiLive Wrote: Hello Friends,

Why the robots.txt file is important ??
Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. Search engines send out tiny programs called “spiders” or “robots” to search your site and bring information back to the search engines so that the pages of your site can be indexed in the search results and found by web users. Your Robots.txt file instructs these programs not to search pages on your site which you designate using a “disallow” command.
Reply
#9
Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Which Important Factors Makes On Page Optimization Better ? swatijain2233 1 16 3 hours ago
Last Post: adamwayne1
  Which is the most important thing for ranking on search engines? ninadordev 2 47 9 hours ago
Last Post: pharmasecure
  Give some important guest blogging sites? vishwa01 7 221 09-10-2018, 11:26 PM
Last Post: RH-Calvin
  Which Is More Important – Bounce Rate Or Exit Rate? ninadordev 11 188 09-10-2018, 04:01 AM
Last Post: vicky
  Why is SEO so important? ZoyaBhatt 17 690 09-10-2018, 01:29 AM
Last Post: sonamg
  Export EDB File To PST carollien 2 73 09-04-2018, 02:55 AM
Last Post: hunzila
  Why SEO is Important for new websites..? ninadordev 2 60 08-31-2018, 10:51 PM
Last Post: sulbha
  Why is AdWords important? ankitasharma123 8 239 08-21-2018, 11:16 PM
Last Post: sastabpo
  How Many Type of Video File..? swatijain2233 4 196 08-03-2018, 06:23 AM
Last Post: EmilyPete
  Why backlink is very important for SEO? canadaimmigrationconsultant 4 214 07-25-2018, 10:11 PM
Last Post: fayeseom

Digg   Delicious   Reddit   Facebook   Twitter   StumbleUpon  


Users browsing this thread:
1 Guest(s)


Webmasterbabble.com, © 2012-2018 .