The History Of Search Engines

When the World Wide Web came into being in the early 1990s, almost twenty years since the 'ARPANET/internet' went public, it has thrown open the doors to a virtual and vast new world of information, which was both unfathomable and ever expanding. But, as the number of web pages and the number of people accessing the internet increased, accessing a particular resource from the sea of information became a problem, especially in the event of a person not knowing the URL of a particular website. Necessity has always been the mother of invention; the need for a tool to search the required resource from the millions of pages of information in the World Wide Web was gravely felt. So what is considered to be the first amongst the first generation of search engines 'Archie' was created. Archie was created by Alan Emtage in 1990 and used the directory listings of all files existing in the public anonymous FTP websites to create a searchable database of file names. A year later, Gopher, which had a similar purpose to Archie, but with text files, was introduced by Mark McCahill at the University of Minnesota.

The programs, Veronica and Jughead, which followed Gopher, worked similarly, on plain text files sent via Gopher. The world of web searches took a decisive turn in 1993, when MIT student Matthew Gray made the first robot, the World Wide Web Wanderer. It was initially meant to count web servers on the World Wide Web, but was later reprogrammed to search URLs as well, thus forming the first website database, Wandex. ALIWEB released in 1993 was also a robot, but with an added feature allowing users to submit their own URLs for indexing purposes.

Web robots are also called web crawlers, web spiders or web wanderers. In its earlier versions, it was criticized for taking too much bandwidth, often leading to a server crash. But the issue was sorted out in later generations. As the World Wide Web started to grow phenomenally by the mid 90s, realizing its financial potential, more players began throwing their hats into the ring.

Excite was the first to make their presence felt and it used statistical analysis of word relations to spot a particular web resource. It was followed by Yahoo in 1994, as a website providing a listing of its favorite web pages. The format of the main link or URL followed by a brief description of what is in the site was first introduced by Yahoo.

Within a year, funds started to flow in, and soon Yahoo became a corporation. Lycos, with its prefix matching, relevance retrieval, and word proximity, announced its entry onto the webi in almost the same period as Yahoo. It was a large search engine and by 1996, it had indexed over 60 million web pages, the biggest by any search engine of that time. Alta Vista is another search tool that had made headlines again in the same period - with advanced search options and allowing natural language inquires.

It was also the first to provide multimedia search options to find music files, videos, and pictures. Ironic as it may be, the search engine that is touted as the most popular today, Google, came into existence rather late in the scene in 1997. It used inbound links to rank websites, but the results it produced were widely accepted and in another year's time, it became the most sought after search engine in the world.

Its nearest competitor, MSN Search, and Open Directory, the biggest hand edited catalogue of the web, came into being in 1998. As of now, Google has the greatest number of patrons, followed by Yahoo and MSN. In the near future, as the scene consolidates, the number of search engines that remain in the race may come down to the current top few. But then all will be providing more advanced and diverse features alongside searching, and it is the common netizens who stand to gain from the final outcome. After all, this is how product economics responds to competition! Finally it should be said, that the number of search engines that have come and gone without leaving a trace in between, are many. However, only the most significant ones have been mentioned in this article.

Scott White is a Search Engine Optimization Arizona Company Gain More Traffic, Learn how to rank your website at the top and Increase Web Site Traffic.