.Google.com's John Mueller addressed a concern concerning why Google.com indexes webpages that are actually prohibited coming from crawling by robots.txt and also why the it is actually secure to neglect the relevant Search Console records about those crawls.Crawler Web Traffic To Question Criterion URLs.The person talking to the concern recorded that crawlers were making links to non-existent question criterion URLs (? q= xyz) to webpages along with noindex meta tags that are likewise shut out in robots.txt. What motivated the concern is actually that Google is actually creeping the links to those pages, obtaining obstructed through robots.txt (without noticing a noindex robots meta tag) after that acquiring reported in Google.com Explore Console as "Indexed, though blocked by robots.txt.".The individual asked the following inquiry:." However listed here is actually the major question: why would certainly Google.com mark web pages when they can't also find the material? What's the benefit because?".Google.com's John Mueller confirmed that if they can not crawl the web page they can not find the noindex meta tag. He additionally helps make an appealing acknowledgment of the website: hunt operator, recommending to ignore the results due to the fact that the "ordinary" consumers will not find those outcomes.He created:." Yes, you are actually proper: if our company can't creep the web page, our company can't view the noindex. That stated, if our team can not crawl the webpages, then there is actually certainly not a whole lot for our company to index. So while you might view some of those webpages with a targeted website:- inquiry, the ordinary user will not view them, so I definitely would not fuss over it. Noindex is likewise fine (without robots.txt disallow), it only means the Links will definitely find yourself being crawled (as well as wind up in the Explore Console report for crawled/not catalogued-- neither of these statuses create concerns to the remainder of the website). The vital part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's solution confirms the restrictions in operation the Website: hunt evolved hunt operator for analysis causes. One of those explanations is since it's certainly not hooked up to the routine search index, it's a separate factor completely.Google.com's John Mueller discussed the site search driver in 2021:." The quick solution is that an internet site: query is certainly not indicated to become full, nor utilized for diagnostics functions.A site inquiry is actually a certain kind of search that confines the results to a particular web site. It's essentially just words internet site, a bowel, and then the internet site's domain name.This inquiry restricts the outcomes to a details site. It is actually certainly not suggested to be a comprehensive collection of all the web pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually alright for these sort of conditions where a bot is linking to non-existent web pages that are actually acquiring discovered through Googlebot.3. Links with the noindex tag will produce a "crawled/not listed" entry in Browse Console which those will not have a negative result on the remainder of the internet site.Check out the concern and also respond to on LinkedIn:.Why will Google index web pages when they can not even find the content?Included Image by Shutterstock/Krakenimages. com.