What’s new in search this April? New crawl bots and a new search engine from Google. Information revealed regarding Google SpamBrain.
During April 2023, these are some of the highlights so far:
- New Google Crawler for R&D Tasks
- Search Marketers are Utilising AI More and More
- Google Working on New Search Engine
- Google to Remove Video Thumbnails if Video is Not Primary Content
- More information on Google SpamBrain Revealed
- Brand Searches Don’t Improve your Search Rankings
Read on for our take on recent events in search.
New Google Crawler for R&D Tasks
Posted April 20th on Search Engine Journal
This information was initially shared by Gary Illyes of Google via a LinkedIn post.
Essentially, Google are unburdening their default crawl-bot (Googlebot) from R&D (Research and Development) tasks. Previously, Googlebot would undertake some basic R&D tasks, in addition to its core mission of crawling and indexing the web. With this change, Googlebot will be able to focus more wholly on crawling and indexation, generating the data with which to feed Google’s various ranking algorithms.
The new Google crawl bot is named “GoogleOther”. You can find Google’ updated documentation for this new crawl bot here:
This means that any “Googlebot” rules within your robots.txt file, will no longer block Google from performing research tasks on your pages. To block Google from those actions, you will now have to add “GoogleOther” rules within robots.txt. If you fail to make this alteration, Google will not suddenly start indexing blocked pages, as those will still be covered by the default Googlebot user-agent.
Here are some comments, from Matt G. Southern, posting on Search Engine Journal:
“The main purpose of the new GoogleOther crawler is to take over the non-essential tasks currently performed by Googlebot.
By doing so, Googlebot can now focus solely on building the search index utilized by Google Search.
Meanwhile, GoogleOther will handle other jobs, such as research and development (R&D) crawls, which are not directly related to search indexing.”
Main takeaway: Unless you specifically want to prevent Google from performing R&D tasks using your website’s page-level data, there’s no reason to take any action. This will also hopefully mean that Google are able to crawl pages more quickly and more efficiently.
Search Marketers are Utilising AI More and More
Posted April 20th on SEO Roundtable
Barry Schwartz, writing on Search Engine Roundtable, recently conducted a very interesting survey. This is what Barry had to say:
“I find myself going back to traditional search over asking AI-based search for answers, and I was curious if I am alone or not. So I created a number of polls on Twitter asking if search marketers are using AI-search or traditional search more and if so, are they using ChatGPT, Bing Chat or Google Bard.
In this poll, I asked which do you use more; traditional search (i.e. Google/Bing) or AI chat (i.e. ChatGPT, Bard, Bing Chat)?
I poll received over 1,000 responses and showed that most of you are using traditional search more than you are using AI Chat. 83% are using traditional search and 17% are using AI chat more. I know this does not mean that you are not using both traditional and AI chat at the same time, but still, I wanted to gauge this somehow.”
Here are the results of Barry’s polls:
Most search marketers are still using traditional search more. However, this doesn’t mean that those marketers aren’t using AI-powered research techniques at all. Interestingly, 16% of search marketers are using AI powered tools ‘more’ than they are using traditional search, that is a massive shift.
Most search marketers are still not making broad use of Bing Chat or Google Bard. Whilst that is true, 32% of search marketers are using ChatGPT weekly. This really goes to show how important it is to push out products ahead of the competition, Google have been left in the dust by Open AI. Whilst most users are still not using Bing Chat, Microsoft have a vested interest in ChatGPT, so they’re not out of the running.
Since Barry Schwartz is such an influential figure within SEO circles, this data is very interesting. One of his polls had over 1,000 responses, and none of these polls received fewer than 800 votes. That’s a fairly good sample of ‘clued-up’ search marketer responses.
Main takeaway: Whilst many search marketers are still using traditional search ‘more than’ AI powered tools for research tasks, 32% of search marketers are using ChatGPT daily. That’s a large shift. Expect AI elements to feature more within your search campaigns. This is the future.
Google Working on New Search Engine
Posted April 16th on Search Engine Land
At first, we thought this might be an April Fool’s post. However, this was written by Barry Schwartz and April Fool’s Day falls on 1st April, rather than 16th April. The initial source for this information was the New York Times, however they are pay-walling their articles. Instead, defer to this post from The Verge:
James Vincent, posting on The Verge, comments:
“The plans are part of Google’s efforts to meet the threat posed by new systems like Microsoft’s Bing chatbot and OpenAI’s ChatGPT. Many think these chatbots could one day replace traditional search engines like Google — despite their failings.
According to the NYT, Google’s position is so threatened that Samsung is considering replacing Google with Bing as the default search engine on its mobile devices. This deal is worth an estimated $3 billion in annual revenue to Google (the company has a similar deal with Apple worth roughly $20 billion), though it’s not clear how seriously Samsung is considering the switch. The company may have been been swayed by Microsoft’s AI work, but it might also be simply taking advantage of Google’s moment of weakness.”
This is extraordinary news. Clearly, Google are one step behind the combined might of Open AI and Microsoft’s search-AI partnership. Going back to Barry’s polls in the previous section of this article, it’s not hard to see why. First, marketers and ‘power users’ adopt new technology. Following this, that information cascades down to your average user, and their technical research abilities improve.
With 32% of marketers using ChatGPT weekly, and most saying that they don’t use Google Bard, it’s clear that Google have been left behind. They have to do something huge to catch up, and their announcement of a new AI powered search engine (which blends search and AI as a more integrated experience) is not wholly unexpected. It’s probably the best they can offer.
Main takeaway: Whilst Google are making huge announcements (such as the development of a new AI powered search engine), they have already been left behind by Open AI and Microsoft’s partnership. Google are also suffering re: Google Analytics 4’s teething issues, and are also potentially being outpaced on the analytics front. Google are giants of search and analytics, but they have not kept up with AI research and developments. Will Google remain a household name in the future? Only time will tell. Search will always exist; people will always search for information. Maybe in the future, most marketers will be operating Microsoft / Open AI Ads, rather than Google Ads, injecting their products into AI chat queries. We, as marketers will survive. Will Google? These truly are unprecedented times.
Google to Remove Video Thumbnails if Video is Not Primary Content
Posted April 14th on SEO Roundtable
Google’s official announcement can be found here.
Google previously rendered video thumbnails in two different ways. Where a search listing connected with a page, and an embedded video was the main content of that page – nothing changes. Google previously allowed video thumbnails, even if the video content was not the primary content, though the video thumbnail was slightly (visually) demoted. These secondary video thumbnail deployments will no longer appear in search listings.
This means that, if you want your search listings to contain a video thumbnail (at all), the video must now be the ‘primary’ content of the connected web-page. If the video is secondary or tertiary content, then it will receive no video thumbnail within search listings.
Here is some commentary from Barry Schwartz, posting on Search Engine Roundtable:
“Google announced it will remove the video thumbnail image from the search results snippet when that video is not the main content of the page. Google said this change will impact search appearance reported metrics for videos in the performance report in Google Search Console.”
So, if your web-page contains video schema for rich snippets, you may see the lost of video thumbnails for some search listings. This doesn’t mean there’s anything wrong with your video schema, and there is no replacement video schema to get these results back again. Instead, you must re-engineer the contents of your web-pages, so that embedded videos are the ‘primary’ content. If you can’t do this, prepare to lose some visibility.
Main takeaway: Google’s search listing rich-snippets will only render video thumbnails, if the video is the main content of the associated webpage. If you lose visibility due to this change, no amount of tweaking your video schema will fix this. Instead, you must alter your pages to promote embedded videos as the ‘main’ page content.
More information on Google SpamBrain Revealed
Posted April 12th on Search Engine Journal
Much of this information is derived from Google’s annual “Webspam Report”, 2022. Each year, Google release this report (looking back over the previous year) and reveal new methods and technologies for catching (and demoting) web spam content.
Roger Montti, posting on Search Engine Journal, had this to say:
“Machine learning is a form of artificial intelligence that uses data to learn to become increasingly proficient at the task it is designed to complete.
Not much is known about SpamBrain other than it’s a machine learning platform and it’s “central” to Google’s initiatives to keep spam from ranking.
The report noted that special link spam training resulted in catching fifty times more sites creating link spam as compared from the year before, citing SpamBrain’s ability to learn as key to its success.
An interesting fact about SpamBrain is how it identifies spam at the time of crawling.
If a crawled page is detected to be spam it is immediately blocked, preventing it from entering Google’s search index and saving resources from being wasted crawling unwanted webpages.
Something new for SpamBrain is that the scam identification system is now multilingual, reducing clicks on scam sites by 50% when compared to the year before.”
This cements a fundamental shift from Google, away from external metrics (such as backlinks) towards AI-evaluation to highlight spam techniques. Of course, as noted, Google still uses backlink data. That data, though, is now subject to AI (rather than flat algorithmic) scrutiny. In addition to this, AI can now critically evaluate pages on a page-level, content basis. If a page appears to be spammy, it can be instantly blocked, without having to defer to backlink metrics.
Main takeaway: SpamBrain’s continued development represents a major evolution for Google’s spam detection abilities. Google will be able to critically evaluate both on-page content, and backlink profiles, in a much more nuanced and accurate way. All this, without having to pay for a major expansion of their manual spam-detection team. This makes it much more important to deploy helpful, meaningful content at all levels. In particular, deploying recycled content to evade duplicate spam penalties, will likely no longer be viable. As ever, the core ingredient for long-term successful SEO, is effort. There is no magic bullet solution.
Brand Searches Don’t Improve your Search Rankings
Posted April 7th on SEO Roundtable
After all this complex AI-centric discussion, let’s finish with something simpler. When users search your brand more, this does not lead to elevated SEO authority for your website, or boons for your non-brand search engine rankings. This is something which any SEO expert could have told you, however it is also a slightly misleading statement. Let’s explore.
Writing on SEO Roundtable, here’s what Barry Schwartz had to say:
“Google’s John Mueller said people searching for your brand name and a keyword phrase does not help you rank better in Google Search or perform better SEO-wise. So having people search for your brand name followed by a keyword phrase won’t help you rank better.
I know this is probably obvious to some, but sometimes I find questions being asked that seem new to me and I like to cover them. The question was worded, “Do branded queries, like + [brand name], help a website rank for non-branded queries?”
John Mueller replied on Mastodon saying, “No, I don’t think that would make much sense.”
It is just too easy to spam, if you think about it. I believe this is how folks use to spam Google Suggest, auto-complete, in the old days.”
Here’s the thread on Mastodon, containing insight from John Mueller, Google’s Search Advocate:
You can see why users might think this. Maybe when users search for your brand more, it leads Google to think that your brand ins more important, thus artificially inflating SEO authority metrics (and leading to higher search rankings elsewhere). What we can tell you, though, is that this kind of ADHOC use of top-level data, is not something which Google usually stands behind. Due to the fact that Google’s many interwoven search algorithms are so complex, applying top-level simplistic overrides often does not lead to ‘better’ search results for Google’s end users.
Google’s search listings are extremely complex. There are numerus data inputs, ranking algorithms, and output search elements to contend with. Google have a propensity to take a more measured, scientific view of their search inputs and outputs. Usually, tests are much smaller and much more pragmatic, with the exception of the integration of ‘game changing’ technologies (such as AI).
So, obviously, Google wouldn’t be utilising such top-level data to interact with their ranking algorithms on such a fundamental, broad-brush level. Whilst it’s easy to see why search users might make this assumption, it’s a red herring.
Why did we allude to the fact that these responses are potentially misleading? Well, one motivation to drive more brand searches might be to increase search performance via increased Google rankings. Whilst we know that higher volumes of brand searches won’t impact your other search rankings, this doesn’t mean that there’s no overall benefit.
Most sites gain the majority of their traffic via branded search terms. If you increase the number of users searching for your brand, you will increase the amount of targeted organic (SEO) search traffic hitting your site. Whilst this won’t have a knock-on effect on your non-brand (or partially branded) search terms, it’s still beneficial. Making people aware of your brand and triggering them to search for your web properties, can be no bad thing.
Main takeaway: When users search for your brand, it doesn’t artificially inflate your non-brand or partially-branded search rankings. Whilst that is true, such techniques still drive highly targeted (brand-term) traffic to your website, which has the potential to convert. Whilst Sina Afkham’s technical hypothesis was inaccurate, that doesn’t make this a bad technique to employ.
Here at Anicca, we stay up-to-**** on all SEO, search and algorithmic developments and apply them to our clients’ campaigns. You can check out our SEO services here. Impressed with our knowledge and coverage of search? Give us a call on 0116 298 7857.
