The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →A web search engine is software that helps you find relevant information on the Web. It searches an organized collection of information about online resources, then selects and orders results in response to your query. Many engines build that collection by crawling and indexing web pages, but they do not necessarily contain every page online.
What is a web search engine?
A web search engine is an information-retrieval system for finding material on the World Wide Web. You enter a query—such as words, a question, or a phrase—and the engine returns web pages or other indexed resources it considers relevant. MDN describes a search engine as software that collects information from the Web and presents it to people looking for something specific: MDN Web Docs’ search engine glossary.
As an Amazon Associate I earn from qualifying purchases.
The engine does not usually inspect the entire live Web from scratch each time you search. Instead, many services maintain an organized collection, or index, of information about resources they have found. A query is matched against that collection to produce results. The Stanford Encyclopedia of Philosophy describes this general category as an information-retrieval system that lets users search distributed digital text: Stanford Encyclopedia of Philosophy’s entry on search engines and ethics.
How does a search engine work?
Google describes three stages for Google Search: crawling, indexing, and serving search results. Other engines may use different designs, but these stages are a useful way to understand the process. See Google Search Central’s guide to how Google Search works.
#1 Best Overall
1. Crawling discovers and fetches pages
A crawler is automated software that visits web addresses and fetches content. It can discover pages through links from pages it already knows about, as well as through other discovery methods. Crawling means that the engine has found and requested a page; it does not mean the page has been added to the index.
2. Indexing analyzes and stores information
During indexing, an engine analyzes information about a fetched page and may store it in an index so it can be retrieved later. That information can include the page’s text and other signals used to understand its content. A page that has been crawled may still not be indexed.
3. Serving selects and orders results
When you search, the engine retrieves resources from its index and orders the results according to how relevant it considers them to be for your query. A page can be in an index without appearing for a particular search: selection and ranking happen in response to the query.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Does a search engine search the whole internet?
No single search should be assumed to cover every page or resource on the internet. Web search engines work with collections they have discovered and processed, and the stages are not guaranteed for every page. Google says it does not guarantee that it will crawl, index, or serve a page, even when the page follows its Search Essentials. A page may therefore be undiscovered, not crawled, crawled but not indexed, or indexed but not selected for a particular query.
Rank #3
That distinction helps explain why a page can be available on a website but absent from search results. Finding a page, storing information about it, and returning it for a specific query are separate steps.
What are vertical and metasearch engines?
“Search engine” can describe systems with different scopes and sources. Two useful distinctions are what a system searches and where its results come from.
Rank #4
| Type | Scope or source | What that means |
|---|---|---|
| General-purpose web search engine | A broad collection of Web resources | Searches across many subjects rather than focusing on one topic or format. |
| Vertical search engine | A limited topic, medium, region, language, or other area | Restricts the search scope; that narrower focus can support deeper coverage in its area. |
| Metasearch engine | Results from multiple search engines | Draws results from more than one service and combines or reranks them, rather than relying only on its own index. |
These categories describe different design choices, not a single universal architecture. The Stanford Encyclopedia of Philosophy discusses vertical and metasearch systems in its overview of search engines.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why crawling, indexing, and results are different
It is useful to treat the stages as separate because each answers a different question: Did the engine find and fetch the page? Did it analyze and store information about it? Did it consider the page relevant enough to return for this query? A “yes” at one stage does not guarantee a “yes” at the next.
For a deeper technical account of crawlers and indexes, Stanford’s Introduction to Information Retrieval chapter on Web crawling and indexes explains the information-retrieval foundations.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
