“Googlebot” have you heard about it? Do you know how Googlebots works? What are the different Googlebot types we have? If you don’t, not to worry, we will discuss all the
Whenever we talk about bots then it reminded us something or someone who came to visit our content on regular basis. A bot is a set of codes which have proper instructions to analyze contents. A bot can access Website SEO, videos, images, texts, links, backlinks, broken-links, Negative Backlinks and many more such things. these things help Google to determine DA, PA and other factors of your site.
Google uses these user agents to crawl any site and their contents. If you don’t want anything to be crawled then just disallow that in the robots.txt file.
Googlebot types and their special purpose
These are the several Google bots which are commonly used. Have a look at them and check your robots.txt to identify are you using them or not?
Googlebot – It crawls pages from Google web index and news index for desktop and smartphone both.
Googlebot-Mobile – It crawls pages for mobile indexing. You might have heard about mobile indexing first.
Googlebot-Video – It crawl pages for video indexing. It also works with YouTube searching. If you use proper tags, titles
Googlebot-Images – It crawls the pages for image indexing. As you know images are an important part of blog content. Without adding an image or video in the post it looks incomplete. So, every image of yours must be indexed properly. Learn more here about Image SEO and Image optimization.
Mediapartners-Google – This is only applicable if you use AdSense on your site. It crawls pages to determine Adsense content. Like when you apply for AdSense, this bot will come to crawl your content like it is ready for Adsense ads or not? People are getting Valuable Inventory: Scraped Content and Valuable Inventory: Under construction AdSense policy violation when they apply for AdSense.
Adsbot-Google – This bot will crawl pages to measure AdWords landing page quality. It is also applicable in the case of AdSense use.
Googlebot-News -This mainly crawls news contents, If you don’t want to use it just use Googlebot, it is applicable for all
There are some other types of Googlebots are also available like Adsbot-Google-Mobile and Adsbot-Google-Mobile-Apps which you can find here.
How Googlebot accesses your site?
For most sites, Googlebot shouldn’t access your site more than once every few seconds on average. Could you believe that? I don’t. But, yes it is real. However, due to network delays, It’s possible that the rate will be seen slightly higher than they say
Googlebot was designed to be distributed on several machines to improve performance and scale as the web grows. Also to cut down your bandwidth usage Googlebot crawl your site from the nearest crawlers available. Therefore, your logs may show visits from several machines at google.com, all with the user-agent Googlebot.
Google’s goal is to crawl as many pages from your site as they can on each visit without overwhelming your server’s bandwidth. You can also Request a change in the crawl rate.
How can you block Googlebot to access content on your site?
Googlebots only accesses and crawls those contents which you allow them to do. This allows and disallows matters should be written in the
For Example – The sample code from a robots.txt file given below is just allowing all Googlebots to access all your content. It is not disallowing anything. This is not only allowing Googlebot it also allow Bingbot, Yandexbot, Ninjabot and other bots available on
Whereas, If you see this sample robots.txt code from our website. You can clearly see the differences. Most of the Googlebots are used and they allowed to access all (Allow:/) contents. But some of the core files are disallowed including trackbacks, comments feed
Disallow: /recommends/ (Affiliate links are disallowed)
Allow: /wp-content/uploads (Only we are allowing some specific folders for image indexing)
Note: If you want to prevent Googlebot from crawling content on your site, just allow or disallow them in your robots.txt file. Be aware of the difference between preventing Googlebot from crawling a page, preventing Googlebot from indexing a page, and preventing a page from being accessible at all by both crawlers or users.
One thing you have to set up in your head if a page is inaccessible for crawl it is automatically inaccessible for indexing as well.
If you have any query or suggestions feel free to comment down.