It is no good having a lot of information but without any way to find them. The internet is an endless maze of billions of documents and resources. Wading through this maze alone would have been an impossibility, without the searching and categorizing services. A search engine is a program designed to search for pages related to a user query, and present pages relevant to the query in a listing. The user query is also called a ‘keyword’. The search engines use a number of methods and parameters to index and rank the web pages on the internet.
The earliest search engine was ‘Wandex’, way back in 1993. The first truly crawler-based search engine was WebCrawler (1994) which let users search for any word within the document. Search engines usually go through the following three processes for any search: firstly, it crawls all or as much of the web as possible; then it makes a list of all the pages, against search terms relevant to the pages; finally it presents the pages in some ranking order when queried. When a user enters a search term on a search engine, it looks up its index of pages against the keyword, and presents the ensuing list in some ranked order – the list contains information regarding the pages, like their page titles, descriptions, and first lines of the pages, etc. Most search engines support the use of the Boolean terms AND, OR and NOT to further refine the search query.
The usefulness of a search engine depends on the relevance of the results it gives back. While there may be millions of Web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Through application of their different algorithms, the search engines try to arrive at some ranking wherein they try to put the ‘best’ pages in the front; while the definition of ‘best’ would differ from one engine to another, each engine tries to give more weight to more authoritative and popular sites. As the search algorithms differ between engines, so do the results for the same search term. The methods also change over time as Internet usage changes and new techniques evolve. It was Google that came up with a clever method to rank pages based on backlink popularity (how many pages link to the page in question) and called it PageRank algorithm. Prior to PageRank, many onsite parameters like meta-tags, keywords density, etc were used for ranking. As the web exists and functions now, the search engines are absolutely necessary for us to find our way.
In this section of the webdirectory, we provide links to many specialized and generic search engines and directories.