1
The robots.txt is a text file which has .txt extension. It is used to tell search engines (like Google, Yahoo and Bing) about the web pages that it should not be crawled and indexed and prevent search engines to crawl to specific area like feeds,trackbacks for security reason, wordpress admin pages and avoid duplicate content because that web pages are not so important for you.
How to create robots.txt file
i) Make a new text document (notepad).
ii) Write your robots.txt.
For example : ( Wordpress CMS )
User-Agent : *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /search?
Disallow:
Save as robots.txt
Note :- upload this file to the root directory of your website.Now you can check Your Site URL.
http://seolearn.in/robots.txt
REPLY TO THIS CLASSIFIED
FEATURED CLASSIFIEDS
Enroll Your Little One Today: Kindergarten Admissions Started !
Why Are Nursery Schools Significant?
American University Of Ras Al Khaimah
H
Have You Ever Noticed Why Indian Schools Outdo Other Schools?
E
Elevate Your Child's Education: Find The Best Private Middle Schools In Sharjah, UAE Today!
What Pointers Do You Need To Look At With Regards To Choosing Schools In Dubai?