Google Admits Crawling Misconception – Implications for SEO

In the Search Off the Record podcast, Google’s Analyst Gary Illyes recently explained how Google-bot treats links during crawling.

His insight disapproves the common opinion that Google-bot crawls websites link by link in real-time.

Illyes’ opinion is that the Google-bot collects links for further analysis rather than sequentially visiting them.

This misconception arises from Google’s documentation of the function.

Contradictory Information

“It is my pet peeve,” Illyes said in the podcast regarding Google’s support pages.

He continues:

“On our site, we keep saying Google-bot is following links, but no, it’s not following links. It’s collecting links, and then it goes back to those links. ”

What The Documents Say

Google’s official documentation on crawler states (emphasis mine):

“Crawler, ” also referred to as a ‘robot’ or a ‘spider,’ is a broad term that refers to any program used in the discovery and crawling of websites through the use of links from one web page to another. ”

The document suggests that Google-Bot crawls through the web through links in real time.

This shows that there is a gap between what Google has been telling the public about its crawler and what the actual crawler does, raising questions about other similar issues that may be circulating in the SEO community.

How this affects the SEO

This revelation has several potential implications for how we understand Google’s crawling process. If this is indeed true, it may alter our views on crawl budgets. Perhaps the collection phase, which is the first step in the described process, could be less costly than initially assumed.  

The notion that Googlebot has to ‘click-through’ to deep pages may no longer exist. This could affect the internal linking strategies and site depth. This insight might also help to explain why some pages are crawled more often than others, irrespective of their rank in the site’s structure.