How search engines work - The basics

Search engines rely on their ability to display relevant search results, the harder they are to manipulate the more accurate they are deemed and the more users they attract. Search engines read, evaluate and index web pages to ensure swift search results, websites need to be indexed because if they weren’t then the search engine would have to crawl every website simultaneously on every search which would not be practical.

The majority of good search engines use the following process to achieve relevant results.

Reading the website content

Websites are read by the search engines complex algorithms, some of these algorithms are known as spiders or bots and they are responsible for reading and evaluating each website.  This process is commonly referred to as crawling.

These algorithms check the content of websites to look for changes and new web pages and have been programmed with a pre-designated objective. This objective dictates the extent and frequency of the crawls dependent on the relevance of the website.

For example a website that has frequent changes and is recognised as significant or valuable is likely to be crawled more intensely and frequently than a small website of low value.

Each crawl records a list of URLs this list is review on each subsequent crawl so as a website changes and grows the spider is able to keep a relatively current directory of the website.

Indexing a web page

When a spider crawls a website it creates an index of each web page, it catalogues the content, the meta tags and the alt tags and stores  the information in a library.  By cataloguing this information it can summarise the content of the web page and evaluate its relevance in future search results.

The current indexed webpage can be viewed by selecting the cached link shown next to the URL of the website in the search results.

The search results

The sole aim of a successful search engine is to deliver timely, accurate, relevant search results, it does this by using the indexed information and analyses the various evaluation results.

Once the most ‘relevant’ web pages have been established the results are displayed in their order of relevance in the form of the websites URL (which would link to the current version of the website not the cache) and the most relevant meta tags from cached website.