If you were trying to find out the best robots.txt file for your WordPress blog then you are at the right place.
Robots.txt is a way to stop Search engine bots from crawling a specific part of your WordPress blog or any site for that matter. The “Disallow” tag restricts a particular file or folder from being crawled by search engines and also other bots. From the perspective of SEO robots.txt file plays a major role.
This is useful to help maintain privacy of your blog content which you don’t want it to be indexed and found in the search index of Google and other search engines. Below are some of the Robots.txt from popular sites and of course NPXP as well. If you can see clearly all of the blogs have made sure that WP folders and files are not crawled by Google and alike.
As you are in need of the best robots.txt file for your WordPress blog, here are some of the best examples that you can refer to and come up with your own version for your blog or site.
Labnol.org
Sitemap: http://www.labnol.org/sitemap.xml
User-Agent: *
Disallow: /wp/wp-admin/
Disallow: /wp/wp-includes/
Disallow: /wp/wp-content/
Disallow: /wp/wp-
Disallow: /go/
Disallow: /translate.php
Dailyblogtips.com
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
ShoutMeLoud.com
NPXP.com
Sitemap: http://www.NpXp.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /archives/
Disallow: /wp-*
Disallow: /author
Disallow: /comments/feed/
Disallow: /trackback/
Sitemap.xml is mentioned exclusively for the purpose of assumed auto discovery by search engines. It is not quite necessary to have one but it doesn’t hurt either if you do.
So NPXP has one in its robots.txt file.
How to use robots.txt file:
Once you come up with your own version or a copied version of Robots.txt file you just need to upload the notepad into the root folder of your blog. So it should be something like this:
No related posts.
Hey, this is Kamal and I operate the NPXP network of websites. I am a blogger as well as a freelancer.
Hi, thanks for the information, On search result of my site, you will find that TAG is also listed by google and for most blog tag is not listed by google, can you please tell me how to do that with the help of robot.txt or by any other method?
Thanks
Rajesh
Hi Rajesh,
You can just do what I have done here: http://npxp.com/robots.txt to blog tags pages from getting indexed.
Code 500 instead of sending robots.txt could block google robot for good with no access error…
Superb web site, exactly where did you found the information in this specific posting? I am pleased I discovered it however, ill be checking back again shortly to discover what other articles you’ve got.
u have mentioned name of shoutmeloud but i cant se its robots.txt file
N 1 more point
ur commentluv doesnt work .. any issues with it??
ATUL