How does Google crawler see pages?
Google crawlers looks the page from top to bottom. However google bot does not sees pages exactly as humans do because it does not render them with CSS or execute JavaScript. Google bot looks and analysis the content of the page and tries to decide the purpose of page. Google bots looks at other signals the page is providing such as robot.txt file which tells googlebot which page is allowed to crawl.
You can prevent pages from Googlebot crawling using robot.txt file
- pages with duplicate content
- private pages
- URLs with query parameters
- pages with thin content
- test pages
Let us see how google bot works:
- The first thing googlebot sees in page is <!DOCTYPE> declaration which tells google bot about version of HTML
- Next it will see the html tag in the page it might also have language attribute. This helps Googlebot to understand the content and provide relevant results
- After that googlebot will look at head tag which contains title which is not shown to users and then meta description tag which defines short summary of the page that may appear in the search results.
- The <head> tag may also contain links to external resources, such as stylesheets, scripts, icons, and fonts, that affect how the page looks and behaves
- The <body> tag may have various elements that structure and format the content, such as headings (<h1>, <h2>, etc.), paragraphs (<p>), lists (<ul>, <ol>, etc.), tables (<table>), images (<img>), links (<a>), forms (<form>), and more.
For example:
Googlebot may use headings to identify the main topics of the page, images to enhance the visual appeal of the page, and links to discover new pages to crawl. After that it will check the closing head tag
How does Google Search Works: Crawling, Indexing, Ranking and Serving
Google is the most used search engine in the world. It contains billions of pages in different categories. Also, new pages are added continuously. Google discovers, crawls, and serves web pages through a complex and automated process that involves several steps. Well, it happens through four main processes: crawling, indexing, ranking, and serving.
Important Topics for How the Google Search Works
- What is Crawling in SEO?
- How does crawling works?
- How does Google crawler see pages?
- What influences the crawler’s behavior?
- What is Indexing in SEO?
- Indexing: How Google Organizes Web Pages
- What is Ranking in SEO?
- Ranking: How are URLs ranked by search engines?
- Serving: How Google Shows Web Pages
- Frequently Asked Questions (Faqs)
- Conclusion