Google Introduces New Crawlers for Image and Video Data Collection

Google has added two new crawlers, GoogleOther-Image and GoogleOther-Video, to its Googlebot family. These crawlers are designed to help with collecting image and video data for research and development purposes.

Details of the New Crawlers:

GoogleOther-Image:

• This crawler is specialized in fetching image URLs from the web.
• It uses the user agent strings GoogleOther-Image and GoogleOther, with the full string being GoogleOther-Image/1.0.

GoogleOther-Video:

• This crawler focuses on fetching video URLs.
• It uses the user agent strings GoogleOther-Video and GoogleOther, with the full string being GoogleOther-Video/1.0.

Purpose of the New Crawlers:

Google introduced these new crawlers to better handle the collection of binary data, like images and videos, which are essential for their research and development projects.

Types of Google Crawlers:

1. Googlebot: The main crawler used for Google’s search engine. It always follows the rules set in the robots.txt file.
2. Special-case crawlers: These perform specific tasks (e.g., AdsBot) and may not always follow robots.txt rules.
3. User-triggered fetchers: These are initiated by users, like Google Site Verifier, which fetches pages based on user actions in tools like Google Search Console.

IP Address Ranges and DNS:

• Googlebot: Uses IP ranges listed in googlebot.json (addresses like crawl-–––.googlebot.com or geo-crawl-–––.geo.googlebot.com).
• Special-case crawlers: Listed in special-crawlers.json (addresses like rate-limited-proxy-–––.google.com).
• User-triggered fetchers: Listed in user-triggered-fetchers.json (addresses like –––.gae.googleusercontent.com).

Why This Matters:

If you monitor bot activity on your website, you might notice these new GoogleOther crawlers. Don’t worry; they are legitimate Googlebots designed to help with collecting image and video data.