What is Robot Text File : -
For those who are new to the term robot text file the simple definition of robot text file is : -
Robot text file is a guide for the search engine bots which are called crawlers and spiders. These bots are assigned a job by search engines to index every site on the internet.
Now as i mentioned above that the robot text file is a guide for these bots. What kind of guide you will ask. The answer is that it's a guide that will force the bots to crawl your site according to your custom needs.
Why Robot Text File is Used : -
Robots text file is used to restrict the crawlers and spiders to index some specific pages or directories in your site that you don't want to get indexed by any search engine or a particular search engine.
For example if you don't want the "About me " and "Contact us " page of your site to get indexed by search engines you can do it by robots text file.
If you are asking yourself should i use robot text file???
Then keep one thing in mind it totally depends on you e.g :- if there is any kind of information on your site that you don't want the search engine to show.
Then its is recommended that you do use this thing. Below in the post i will be showing you how you can write one for your site it's quite simple. But if you want the crawlers to index every thing in your site then i recommend you to leave it blank.
How to Write Robots Text File : -
There are few things that i will let you know that are essential to write one for your site.Just check the definitions given below first. I will give a default robots text file deep in the post so if you don't want to create one you can use that one and skip this part of the post.
User-Agent : These two words are used instead of search engine.Simply it means search engines.
* (star):
That star or what ever you want to call it means "all "., e.g: if i use it in front of user-agent like this " user-agent: * " .It means that i am writing the command for all of the search engines.
That star or what ever you want to call it means "all "., e.g: if i use it in front of user-agent like this " user-agent: * " .It means that i am writing the command for all of the search engines.
Disallow : This simply means that you want something that should not be indexed by the bots.
/ (forward slash):
This forward slash should be used next to disallow.If you put the forward next to the disallow command like this " Disallow : / " it means that you are saying to the bots to that don't index the whole site. Don't do that.Now what you should be doing is, if you want temp directory of your site not to get indexed by the bots than the command should be like this " Disallow : /temp " that will cause the bots to leave the temp directory of your site.
This forward slash should be used next to disallow.If you put the forward next to the disallow command like this " Disallow : / " it means that you are saying to the bots to that don't index the whole site. Don't do that.Now what you should be doing is, if you want temp directory of your site not to get indexed by the bots than the command should be like this " Disallow : /temp " that will cause the bots to leave the temp directory of your site.
Now that you have learned the above terms you can write one of your own. I have a given a default robots text file you can modify it and then use it for your site or if you don't want any changes just simply use the code i have given below.
Recommended post >> How to Enable Meta-description for each Post
Recommended post >> How to Enable Meta-description For blogger
How to add Robots Text File in Blogger : -
1.Go to your blogger dashboard and select your blog.
2.Now click on setting then search preferences.
3.Now take a look on the Crawlers and Indexing Section then click on edit Custom Robots.txt
4.Then you will find a box similar to the one given below
Now paste this code in the box or make any other changes you want in it.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://eliteblogpress.blogspot.com/feeds/posts/default?orderby=UPDATED
Note : Don't forget to replace the url of your blog written in red color.
Share your Views : -
Do tell us what are your views about this post.You can share your views simply by commenting below. If you have any query about this post than contact me i will be available for you and in the mean time if you liked this post so share it to help others like you. More like these will be coming in future so stay in touch.
No comments:
Post a Comment