About robots.txt file:

Robots.txt file is often used by many websites for the assurance of their security. However, before you use it, you must know why you need it. This is file is useful, however, if your site is secured from unwanted and unethical attacks like hacking, you may not need it. Robots.txt file save your site from being crawled by the search engine bots.  The search engine crawlers look for special files on your website which tells them about the pages which should be indexed and which shouldn’t be. This is done by the robots.txt file. Without this, it can be that all of your web pages get indexed and appear on the internet. Suppose you have something that is confidential like the personal details of your customers, in case you are a shopping portal, then surely you won’t like to have them shared on the internet. Imagine what it will land you to, if all these details are made accessible to all and sundry on the internet. You and your customer, both are going to face the music.

How you can create one:

This is a simple file which requires you to open a text editor in order to create it. This file consists of records and each record contains of information for a particular search engine. A record also consists of a user agent line and a disallow line.

Therefore a robots.txt file acts as instructor for the search engine bots, telling them which web pages should be indexed and which should be ignored for maintaining the confidentiality of the website.

Are they effective?

These may instruct the bots to avoid certain pages, but this doesn’t imply that the robots.txt files are effective in preventing them from accessing the web pages. Search engine crawlers have in-built artificial intelligence and are capable of visiting any web page in a website and indexing them. However, in doing so, they are also going to face much legal objections, which is why they never do so. Therefore you needn’t worry about the safety of your website from the search engine crawlers.

Other threats that you are sure to face:

While the search engine crawlers are left with no other alternative than accepting what your robots.txt will tell them, there are others who have no respect for this. The ones like spammers and others with intentions to access your private pages and take advantage of it. Of course the bad news is that while you will know of this, you cannot do much about it. You can avail other safety measures like firewalls, password protection and other encrypted methods.

Now do you really need robots.txt file when you have so much of security before? Yes you may. In case you have content which is a lot important and cannot be indexed at any cost, you can use this file in order to have an additional security. If you don’t have anything much to hide, you don’t need to create a robots.txt file, even if it is an empty one.