.Google's John Mueller responded to an inquiry concerning why Google marks web pages that are refused from creeping by robots.txt and why the it's risk-free to ignore the associated Explore Console reports regarding those creeps.Crawler Traffic To Question Parameter URLs.The individual inquiring the inquiry recorded that bots were producing links to non-existent query parameter Links (? q= xyz) to pages with noindex meta tags that are actually additionally blocked out in robots.txt. What cued the concern is actually that Google.com is actually creeping the hyperlinks to those webpages, receiving shut out through robots.txt (without envisioning a noindex robots meta tag) at that point getting reported in Google.com Browse Console as "Indexed, though blocked through robots.txt.".The person talked to the following concern:." But below is actually the big question: why will Google.com mark webpages when they can't also find the information? What is actually the advantage during that?".Google's John Mueller affirmed that if they can't crawl the web page they can't find the noindex meta tag. He also produces an intriguing mention of the internet site: search driver, advising to overlook the results because the "typical" consumers will not observe those outcomes.He created:." Yes, you are actually appropriate: if our team can not creep the page, our company can not see the noindex. That pointed out, if our team can't crawl the web pages, at that point there is actually not a lot for us to index. Therefore while you could observe a number of those webpages along with a targeted website:- question, the average consumer will not view them, so I wouldn't bother it. Noindex is actually likewise fine (without robots.txt disallow), it simply implies the URLs are going to end up being actually crept (as well as end up in the Look Console report for crawled/not recorded-- neither of these conditions create issues to the remainder of the site). The fundamental part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the constraints in operation the Web site: hunt evolved search driver for diagnostic main reasons. Some of those explanations is considering that it's not linked to the normal search mark, it's a separate point altogether.Google's John Mueller talked about the site hunt driver in 2021:." The short answer is that an internet site: question is certainly not meant to become complete, nor made use of for diagnostics objectives.A website question is a certain type of hunt that restricts the end results to a particular site. It's generally only words site, a digestive tract, and after that the site's domain name.This query restricts the end results to a particular website. It is actually certainly not meant to be a complete compilation of all the pages from that web site.".2. Noindex tag without utilizing a robots.txt is great for these type of situations where a bot is actually connecting to non-existent webpages that are obtaining discovered through Googlebot.3. URLs with the noindex tag will certainly produce a "crawled/not indexed" entry in Browse Console and that those will not have a bad effect on the remainder of the site.Review the inquiry as well as answer on LinkedIn:.Why will Google mark webpages when they can not even observe the content?Included Picture by Shutterstock/Krakenimages. com.