What Are Robots and Crawlers?
Robot, spider, or crawler is a part of software that is programmed to “crawl” from one web page to another search for the links on those pages. It collects content (text, inks) from web sites and saves those in a database that is indexed and ranked according to the search engine algorithm. The links in a crawl will sometimes take the crawler to other pages on the same web site, and sometimes they will take it away from the site completely. The crawler will follow the links again and again until every link on a page has been followed.
The crawler sends a request to the web server where the web site resides for review. The difference between what your browser sees and what the crawler sees is that the crawler is viewing the pages in a completely text interface. No graphics or other types of media files are displayed. It is all text. If the site doesn’t eventually begin to cooperate with the crawler, it’s penalized for the failures and your site’s search engine ranking will fall.
