Welcome to our video explaining how search engines work. We’ll give you the basics on how search engines find web pages, what they do with the pages they find, and how they decide what results to show.
When you’re using a search engine to find the closest café, you’re probably not thinking about search engine technology. But later you might wonder, how did it do that? How did it sort through the entire Internet so quickly, and choose the results you saw on the page?
Each search engine uses its own software programs, but the way they work is pretty similar. They all perform three tasks:
First, they examine the content that they have permission to see (that’s called crawling).
Second, they categorise each piece of content (that’s called indexing).
And third, they decide which content is most useful to the user's query (that’s called ranking).
Let’s take a closer look at how these work.
Search engines “crawl” the Internet to discover content, like web pages, images and videos. Each search engine uses computer programs called “bots” (short for robot), “crawlers” or “spiders” to make its way through the pages.
The bots hop from page to page by following links to other pages. These bots never stop; their sole purpose is to visit and revisit pages looking for new links and new content to include in the index.
So indexing is the second part of the process. As the name suggests, it's very similar to how readers use the index at the back of a book. The index is a gigantic list of all the web pages and content found by the bots. The search engine uses this index as the source of information displayed on the search results pages.
But, not everything the bots find makes it into a search engine’s index.
For example, search engines may find multiple copies of the exact same piece of content, located on different websites.
How is that possible? Well, imagine you’re not searching for a café, but a coffeemaker. You might notice that the top-of-the-line CoffeeKing2000 has the same word-for-word description on the websites of many major retailers. The description might have been provided by the manufacturer... but now the search engine has decisions to make: which version to keep in the index? There’s no need for hundreds of duplicates, so it’s unlikely that every page will be added.
So if you own a website that’s selling coffeemakers, you’re likely better off writing your own description of the CoffeeKing 2000.
That covers crawling and indexing, which just leaves us with ranking. When you type in a search, the engine compares the words and phrases you use to its index, looking for matching results. Let’s say, for example, the search engine finds 230 million matching results. Now it’s time for the last part of the search engine’s task: ranking.
The way search engines rank pages is top secret—it’s their ‘special sauce.’ There are hundreds of ways search engines determine rank, including things like the words on the page, the number of other websites linking to it, and the freshness of the content.
But no matter what formula they use to determine rank, the goal remains the same: to try to connect the user with what they are looking for.
So, what have we learnt? Search engines are constantly working to scour the web for content, organise it and then display the most relevant results to searchers. Understanding this process will help you make your website the best it can be.
- how search engines find web pages
- what they do with the web pages they find
- how they decide what to show on search results pages.