Search is a lookup in a prepared catalog
When you type a query and results appear in a fraction of a second, the engine is not scanning the live web at that moment. That would be far too slow. Instead, it searches a huge catalog it has already prepared. Building and using that catalog happens in three broad stages: crawling, indexing and ranking.
Stage 1: Crawling
A crawler, sometimes called a spider or bot, is a program that visits web pages, reads them and follows the links it finds to reach more pages. Starting from known pages, it keeps discovering new ones through links. Site owners can also help discovery by submitting a sitemap, a file that lists the pages they want found.
Crawlers cannot visit everything all the time. They decide what to fetch and how often, usually based on factors such as how important a site seems, how often it changes, and how quickly its server responds. Busy news sites may be revisited very often, while a rarely changed page might be revisited much less frequently.
Site owners can give instructions through a file called robots.txt, which asks crawlers not to visit certain areas. It works on the honor system: reputable crawlers follow it, but it is not a security tool, and it does not make a page private. Pages that require a login, or that no other page links to, may never be found.
Crawlers retrieve a page the way a browser does, as discussed in the article on web browsers, and many engines also run a page’s scripts so they can see content that is built after loading. This is more work, so content that depends heavily on scripts can be processed later or less reliably than plain text.
Stage 2: Indexing
Once a page is fetched, the engine analyzes it. It works out what the page is about by reading the text, headings, titles, image descriptions and other parts. It also records signals such as the page’s language, when it was last updated and which other pages link to it.
The results are stored in an index. A rough picture is the index at the back of a book. Instead of listing pages and the words on them, it lists words and the pages that contain them, which makes it quick to find every page related to a term.
Not every crawled page is indexed. Engines may skip pages that are near-duplicates of others, that carry little useful content, that are blocked by a “noindex” instruction, or that appear to be spam. When several addresses lead to the same content, the engine often picks one as the preferred version.
Stage 3: Ranking
When you search, the engine first tries to interpret what you mean. It may correct spelling, recognize synonyms and guess whether you want a definition, a local business, a news story or a product. Modern engines use machine learning, including language models, to better connect the meaning of a query to the meaning of a page, rather than only matching the exact words.
It then pulls candidate pages from the index and ranks them. Engines combine a large number of signals, and the exact formulas are not public and change regularly. Commonly discussed factors include:
- Relevance: how well the page’s content matches the query and its likely intent.
- Quality and usefulness: whether the content seems original, accurate and helpful.
- Links: whether other reputable pages link to this one, which is treated as a sign of value. Early search engines made this idea famous.
- Freshness: for some queries, such as news, newer pages matter more.
- Usability: whether a page loads quickly and works well on phones, and whether it is served securely over HTTPS.
- Context: location, language, and sometimes past activity can change what a person sees.
Because of the last point, two people searching the same words can get different results. Results can also change from day to day.
What the results page contains
The page is not just ten blue links. Engines often show featured answers, maps, images, videos, shopping results and related questions. Clearly labeled advertisements also appear, and these are sold through a separate auction-style system. Paid placement is distinct from the organic ranking described above, which cannot be bought directly.
What site owners can influence
The practice of making a site easier for engines to understand is called search engine optimization, or SEO. Sensible basics include clear page titles, helpful original content, working links, fast loading and a site that works on mobile devices. Nobody outside the engine can guarantee a particular position, and claims of guaranteed rankings deserve skepticism. Methods that try to trick the system, like stuffing pages with keywords or buying large numbers of links, can lead to a page being demoted or removed.
Limits worth knowing
- A search engine shows what it has indexed, not “the whole internet.” Large parts of the web are not indexed, including private pages and content behind logins.
- A high ranking is not proof that a page is correct. Ranking systems estimate usefulness; they do not verify facts. Checking sources is still the reader’s job.
- Engines are shaped by the data and choices of the people who build them, so results can reflect biases in what is published and how it is measured.
A short recap of the flow
Crawlers find pages by following links. Indexing organizes what they found. Ranking picks and orders what to show for each query, all within a very short time. Understanding that sequence explains most everyday puzzles, such as why a brand-new page does not appear at once, or why an old page can linger in results after it changes.