.Google.com's John Mueller answered a concern about why Google indexes pages that are prohibited coming from crawling through robots.txt and why the it's risk-free to neglect the similar Browse Console files about those crawls.Crawler Traffic To Query Specification URLs.The individual talking to the concern chronicled that crawlers were making hyperlinks to non-existent query guideline Links (? q= xyz) to pages with noindex meta tags that are actually likewise obstructed in robots.txt. What triggered the question is that Google is crawling the web links to those pages, obtaining blocked out through robots.txt (without noticing a noindex robots meta tag) then acquiring shown up in Google Look Console as "Indexed, though blocked by robots.txt.".The person asked the following question:." However below is actually the huge question: why will Google index pages when they can't also view the web content? What's the conveniences in that?".Google.com's John Mueller affirmed that if they can't creep the page they can't view the noindex meta tag. He likewise helps make an exciting acknowledgment of the website: search operator, encouraging to dismiss the outcomes given that the "ordinary" individuals won't see those end results.He wrote:." Yes, you're correct: if our experts can't crawl the webpage, our company can not view the noindex. That claimed, if we can't crawl the pages, after that there is actually certainly not a whole lot for us to index. Thus while you may find several of those web pages with a targeted website:- question, the ordinary individual will not view all of them, so I definitely would not fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it just indicates the Links are going to find yourself being actually crept (and end up in the Browse Console report for crawled/not recorded-- neither of these statuses trigger concerns to the rest of the web site). The vital part is that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the limits in operation the Website: search advanced search operator for diagnostic causes. Some of those factors is because it is actually certainly not hooked up to the normal search mark, it is actually a separate trait completely.Google's John Mueller commented on the website hunt driver in 2021:." The brief response is that a website: query is certainly not suggested to become complete, neither utilized for diagnostics objectives.A website query is a certain type of hunt that confines the outcomes to a particular web site. It is actually essentially just words internet site, a digestive tract, and afterwards the internet site's domain.This inquiry limits the end results to a specific site. It is actually certainly not indicated to become a detailed assortment of all the web pages from that internet site.".2. Noindex tag without making use of a robots.txt is actually fine for these type of circumstances where a robot is connecting to non-existent pages that are receiving discovered through Googlebot.3. Links along with the noindex tag are going to create a "crawled/not listed" item in Explore Console and that those won't have a bad impact on the remainder of the web site.Read the inquiry and also answer on LinkedIn:.Why will Google index webpages when they can't even find the content?Featured Image by Shutterstock/Krakenimages. com.