Web owners can take a control over crawlers that visit their web site. Instead of using the robots.txt file, you use a robots meta tag to make your preferences known to the crawlers. The robots meta tag is a small piece of HTML code that is inserted into the HEAD tag of your web site and it works generally on the same way that the robots.txt file does. You include your instructions for crawlers inside the tags.
meta name=”robots” content=”noindex, nofollow”
meta name=”description” content=”page description.”
This bit of HTML tells crawlers not to index the content on the site and not to follow the links on the site. The major difference between robots.txt and robots meta tags is that with the meta tags you cannot specify which crawlers you’re targeting. It’s an all or nothing tag, so you either command all of the crawlers to behave in a certain way, or you command none of them. It’s not as precise as robots.txt, but if you don’t have access to your web server, it’s a good alternative. Unfortunately, not all search engines recognize the robots.txt file or the robots meta tags and you have to care about it.
Tags: Meta Tag