This allowed bots to crawl the facet navigation filters Belgium Phone Number List without limits. The result? In 1 month time: were indexed 13.2 million pages of type filter pages that do not contain canonicals 88 million URLs 'excluded from indexing' pages that were Belgium Phone Number List created but not indexed by a present canonical at its peak, 3,559,900 pages were crawled per day with 60 gigabytes of data downloaded by Google bots These are hefty numbers, even for Belgium Phone Number List the biggest online giants. Rankings were thankfully unaffected throughout the process. What was the problem?
The problem was Belgium Phone Number List twofold: The crawl budget went to irrelevant pages A huge amount of irrelevant pages were indexed Crawling pages with crawling tools helped in part: the number of pages involved was simply uncrawlable. The solution: All Belgium Phone Number List links that point to URLs with parameter that do not have condition X, must be provided with a no-follow (on the link) A disallow combined with a noindex in the robots.txt for Belgium Phone Number List URLs consisting of one of 10 types of parameters. Each type of URL parameter got its own disallow and noindex.
This was a 'dirty' solution at the time, but one Belgium Phone Number List that was effective*: bots understood this to unindex the page and stop crawling the page. The tricky part was redirecting the crawl budget to the irrelevant filter parameter URLs that are already indexed. After all, the continuous expansion of pages led to the crawl budget being continuously Belgium Phone Number List allocated to newly created filter combination pages. An interim solution The method finally applied was an interim solution in which the relevant pages were given a noindex at page level Belgium Phone Number List and AJAX functionality was applied to the facet navigation links.