Google’s Gary Illyes said on the last Search Off the Record podcast that Google may crawl certain sections of your site more frequently and also infer the quality of certain sections of your site differently.
It happened at minute 9:09 in the podcast, but Glenn Gabe summed it up very nicely on Twitter. Glenn said “Google can infer from an overall site which areas it might need to crawl more frequently. For example, if there is a blog subdirectory and if there are signals that it is popular/important, then Google might want to explore it further.” “And it’s not only the frequency of updates, it’s also a question of quality. For example, if G sees that a certain model is popular (folder), and people talk about it and are connecting to it, so that’s a signal that people like this directory,” he added.
Here is the embedded video:
Here is the transcription of this section on the frequency of crawls by section of the site:
Yeah. Because, as we said, we don’t have infinite space, so we want to index things that we think of — well, not us — but our algorithms determine that they could be searched at a any given moment, and if we don’t have signals, for example, yet, on a certain site or a certain URL or whatever, then how would we know that we need to crawl that for indexing?
And some things you can infer – for example, if you’re starting a new blog on your main site, for example, and you have a new /blog subdirectory, for example, then we can kind of infer, on the base of the whole site, whether we want to explore a lot of this /blog or not.
So here is the quality section:
But it’s not just the update frequency. It is also the quality signals that the main site has.
So, for example, if we see that a certain pattern is very popular on the Internet, like a slash product is very popular on the Internet, and people on Reddit are talking about it, other sites link to URLs in that pattern, so that’s a signal to us that people like the site in general.
Whereas if you have something that people don’t link to, and you try to start a new directory, it’s like, well, people don’t like the site, so why would we explore this new directory you just launched?
Discussion forum on Twitter.