Optimizing Robots Text File to aid SEO(Search Engine Optimization)

Optimizing robots text file is really important, if it's not done then search engine bots might crawl your website for confidential information and can display them in their respective search engines or if you are using robots.txt file but it's not written properly then it can be one of the scenarios that it might block search engine from crawling the website.

We can block spiders to crawl restricted parts of our website. Restricted parts of our website means those links of our website which we don't want to be indexed in search engines and getting some unwanted visitors. For example :-

How many of you would be interested in indexing your administration page in search engine ?

In past while I have seen some scenarios in which website owners were paranoid in using robots.txt in their website as they were scared that this will harm SEO. Well this is not true , If we can use robots.txt properly then we can stop the crawler from particularly crawling restricted links and it will crawl every other link which we will not restrict and it will not even harm SEO of our website. We can accomplish this task by using robots.txt .

Before we can discuss optimization and can take the full advantage of robots.txt, we should first discuss the basic concepts of robots.txt

A robots.txt is a text file that has to be placed in the root folder of your web server (where you place index page of your website). You can simple create this file in a notepad. It tells various search engine bots that which part of website should not be crawled or should not be indexed. By using this we can instruct bots to prevent our website from being crawled or we can instruct them that they should not crawl or index certain areas of the website. Even we can use same robots.txt to give different instructions to different bots.

Even if you don't want to protect any area of your website from indexing or crawling, still you should use robots.txt as it can act as a open invitation for search engines to crawl your complete website.

There can be several scenarios in which you might be interested in blocking Search Engine Bots from crawling certain parts of your website.
For example

1) Protecting your administration panel of your website.
2) Protecting your under construction pages from getting indexed in search engines.
3) Protecting directory that you don't want to be indexed like cgi-bin
4) Protecting pages that have email addresses as they can be used by spammers if got indexed in search engines.

Reasons can be various but the solution is same , that is using robots.txt now let's start writing in robots.txt

Basic syntax of robots.txt is

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

You can repeat these lines for blocking different directories or giving different instructions to different spiders. Let's get in few examples that will make it more clear. If you don't know the name of any particular Spider or Bot then you can refer here List Of Search Engine Spiders

Example 1) Exclude a file named private.html in private folder from being crawled by Googlebot .

Solution 1) In this scenario you can write the following code in robots.txt

User-Agent: Googlebot
Disallow: /private/private.html

Example 2) Exclude a folder named private from being crawled by search engine.

Solution 2) In this scenario you can write the following code in robots.txt

User-Agent: *
Disallow: /private/

Example 3) Instruct Search Engine bots to crawl and index everything on the website.

Solution 3) In this scenario you can write the following code in robots.txt

User-Agent: *
Disallow:

Example 4) Instruct Search Engine bots that they should not crawl or index any part of the website.

Solution 4) In this scenario you can write the following code in robots.txt

User-Agent: *
Disallow: /

Example 5) Exclude multiple folders (private1,private2,private3) from being crawled by search engines.

Solution 5) In this scenario you can write the following code in robots.txt

User-Agent: *
Disallow: /private1/
Disallow: /private2/
Disallow: /private3/

Example 6) Instruct Googlebot to crawl everything on the website and instruct Alexa bot that it should not crawl any part of the website

Solution 6) In this scenario you can write the following code in robots.txt

User-Agent: Googlebot
Disallow:

User-Agent: Alexa
Disallow: /

I am sure that after reading this article you have got the fair idea about robots.txt and you can now use robots.txt to aid SEO of your site.

Was this helpful? 0 0 Comments

You're reading Optimizing Robots Text File to aid SEO(Search Engine Optimization).